Our Discussion With John “Traenk” Traenkenschuh Continues

From natural disasters such as tornadoes, to (recently declared) pandemics like swine flu, our world practically demands companies protect data with business continuity and disaster recovery plans.  And yet, some do not.

In part 1 of our Data Center Leaders Interview with IT veteran John “Traenk” Traenkenschuh, we discuss how business continuity and disaster recovery plans have evolved and how they may look years from now.

In Part 2 below, John leverages real world insight, including that  gained from his experience as a ‘Wave One’ responder in the wake of  Hurricane Andrew, to discuss the  factors that should compel all companies to develop strong continuity and disaster recovery plans:

Evolving Solutions:
Dan Blacharski of IT World ran a poll asking readers to share the disaster recovery plans they have in place.  As of this writing, 31% of respondents answered ‘backup software with off-site storage.’  However, almost as high a percentage of respondents had no plan.  What do you find most surprising about these results?
(Editor’s Note:  Poll numbers have since updated to 40% ‘backup software with off-site storage, 28% with no plan.)

John “Traenk” Traenkenschuh:
I’m surprised that organizations are discussing, publically, those problems that they are screening privately.

How many organizations must admit that not only are their backup media stolen, the same media are unencrypted as well?  Too many organizations have no BRP plan and associate BRP with system back up only, whether a prioritized list of core applications exists or not.

These organizations are most vulnerable to a massive disaster that breaks operations for hours, if not days.  Too many organizations have no identified BRP coordinator.  Most have never had a practice outage.  Most have no NOC (Network Operations Center) documentation to guide a BRP event.  Just as bad, too many have documentation done without the help of the technical staff.  These plans are full of ‘smoke-and-mirror’ assumptions such as ‘IP just plain happens here’.

Whether swine flu, bankruptcy, regulations, or natural disasters like tornadas (what happens to a tornado that swings though my part of the country); there have never been a more compelling set of factors requiring true BRP expertise.
Sadly, never have our most prominent organizations been least prepared to cope with today’s events.

Don’t believe me?  Read the news.  Find out how much critically important data is kept on laptops.  Experience the horror as a backup media theft has a ripple effect that has hundreds of thousands of customers given public notice that an organization is negligent at best.  A good Business Resumption Plan anticipates these exposures and helps the organization take steps to lessen impacts.

Most concerning are those stories that reveal significant parts of the infrastructure are open to attack.  Surprised by the results of the poll?  Not at all.  Hopeful those will change?  Certainly.

Evolving Solutions:
What factors play most heavily in developing a disaster recovery or business continuity plan, for example, government regulations, client contracts, or something more?

John “Traenk” Traenkenschuh:
Yes.

There is no single one factor that motivates even the busiest organizations.  It is a series of factors, acting randomly, that is creating ‘Perfect Storm’ conditions for compliance.  While you list excellent and compelling factors, we miss some of the more compelling.

Swine flu is a perfect example of an exposure that crops up out of nowhere.  Much like a 1970’s ship disaster movie, the premise seems almost ludicrous.  There is a new flu that can kill (and has).  It incubates in pigs, but leaps onto humanity like plague-filled fleas from centuries ago.  World Healthcare organizations predict a pandemic, in a vain attempt to predict the spread (in a small-world global community full of same-day flights).

When too little happens, organizations take this as a sign of needless panic and assume it’s ok to assume the best.  And then it comes, as assuredly as the ship’s belly-flop, everyone is backpedaling to try to fix the blame–instead of working to fix the problem.

I was a ‘Wave One’ responder after the Hurricane Andrew disaster.  Discussing Disaster Recovery conditions with those who have never helped in a sizeable Disaster Recovery event is so very frustrating.  Much like those BRP plans done without detailed knowledge of the underlying technologies, conversing with inexperienced BRP people is a lesson in futility.

It is rife with unfounded assumptions, those as ridiculous as “IP happens here”.  What happens when your organization calls in BRP workers from far away and your area has most road signs blown away?  When the switched landlines are gone, what mobile and cellular options have you lined up?  The water is yellow and brown; what do you do for your onsite workers, knowing local bottled water was sold out two days before?  You’re counting on local expertise to bridge the documentation gap; they have their own tragedies and family illnesses to work through.  What do you do now?

It is the raw unpredictability of today’s business events that compels us to work through the Disaster Recovery specifics, earlier, versus waiting for later.  Those organizations that aggressively prepare for BRP, with noted and experienced BRP experts, will survive.  Those that do not will find their instance of a Heartland data loss (or flu quarantine or…) may find survival impossible.

Evolving Solutions:
What tips would you offer for a business as it develops disaster recovery or business continuity plan for the first time?

John “Traenk” Traenkenschuh:
Surprisingly the most basic, the simplest advice, is seldom heard.  I would like to review a few basic ideas I share freely.  BRP need not be an overly expensive exercise in paperwork and meaningless reports.

a.     Talk with your business insurance professional – I am always surprised to find a business, no matter how small, that has no business insurance plan.

The Insurance Industry can only profit as your business avoids losses and/or lessens impact.  The right representative will take a personal interest in your business and keeping it free from grave exposures unconsciously assumed.  An onsite inspection, useful when calculating premium, can help find those crowded exit hallways that are littered with combustibles.  The inspections may be done for free in some cases; and if so, this can help you begin BRP efforts with low costs.

At some point, your organization will need to decide which exposures (and resultant costs) you will assume yourselves.  The biggest BRP issue too many face is:

•    Vagueness regarding what risks you still face,
•    Uncertainty concerning which of these you have self-insured, and
•    Indecision and inertia over which loss and impact mitigating steps your organization has signed up for.

b.    Create a BRP team within your organization – Let me guess…  You’ve already identified your ‘Goto Guy’ and have pinned all your hopes (and equal wrath) on him or her.  If this is true for your organization, this is ill advised for both political and for BRP reasons.  BRP is seldom popular when times are good and funding is available.  It becomes ‘hysterical over-reaction’ when times are tight and layoffs are ongoing.  Who in your organization is encouraging and funding BRP in these difficult times?

(Ironically, the downturns and layoffs are the most important times to have BRP plans updated!)

Your team needs to have members from a few influential parts of the organization, to ensure minimal funding at all times.  The same small team also ensures that the loss of any one member keeps BRP event handling continuous.  Seeing BRP as an IT-only exercise is another ‘biggest’ mistake an organization might make.  Until BRP is funded and supported by the organization itself, it simply will never be completed.

c.    Engage the services of a ‘Hired Gun’ – The stakes are high, very high.  The required skills involve IT, security, disaster recovery, virtualization, system and application design, IP networking, etc.  You will need an objective third party to bounce ideas off of.  At some point, the same party will need to coordinate disaster simulation exercises.  There is no reason to create forms and process flows for readiness—there is no time for that.  Implement, and not just plan, your BRP initiative!

Be ready for a series of tough questions that challenge your assumptions regarding how your organization operates.  Process improvements, including a painstaking inspection of externally sourced important applications and infrastructure, will be part of the package you hope to buy.  If there is no appetite for change, there can be none.  Meanwhile, how many of your best, most knowledgeable workers just retired or were laid off?  Change is inevitable despite our rejection or acceptance of it.

Hint:  BRP professionals who cannot work remotely or create suitable remote access systems, who insist on expensive and frequent onsite visits, these same people may be incapable of helping your organization’s flexibility during an actual BRP incident.

d.    Test (and Retest) your BRP technologies – Get the right support from the leadership and simply shut down a critically important system and/or application.  How quickly is control and coordination responsibility (and decision making authority) transferred to the correct group?  Were the promised backups performed reliably and were they available?  Hopefully, you will find that all went as planned.  If not, you will have the means to identify and remedy the problems.

I know that I have over-simplified what needs to go into a BRP plan and readiness effort.  Those organizations that are truly interested in improving their BRP readiness will act on steps this simple.  Others will simply laugh it off as hysteria, until it is too late.

Evolving Solutions:
Wild Card: Anything else you’d like to add?

John “Traenk” Traenkenschuh:
We started this interview with a brief glimpse at the business and IT improvements that are possible with virtualization technology, our second chance to exercise good governance and stewardship over the applications and information placed under our care.  The latter part of the interview discusses Business Resumption Planning and the many ways this field of study has grown, despite lack of interest from those organizations risking public and regulatory ridicule for simple mistakes.

Virtualization can provide focus for BRP efforts, all the while ‘Perfect Storm’ factors call on today’s BRP professionals to be ready for new and more challenging disaster scenarios.  In all of this, the issue is not a technical issue.  Instead, it is an issue with organizational efficiency and ability to face the challenges.  All the advice in this interview is provided freely, with no claims given to its sufficiency or perfection.  These views do not reflect the views of my employers, the organization publishing this interview, or any of my professional organizations with which I am associated.

I promise to do my best to respond to any and all queries left in response to this interview.  BRP, Virtualization, and Green IT are more than slogans to me.  They are a key to surviving in a radically changed business and social environment that seems more the stuff of nightmares than the evening news.