Business Continuity & Disaster Recovery Expert John “Traenk” Traenkenschuh

In a recent interview with Lawrence Webber, we discussed the ‘hows’ and ‘whys’ of business continuity and disaster recovery planning.  This week, our Data Center Leaders Interview Series drives home the importance of this topic during our two part interview with John “Traenk” Traenkenschuh.

Budding author, book editor, and Information Technology worker at three Fortune 100 companies, John “Traenk” Traenkenschuh’s insight into business continuity and disaster recovery planning comes courtesy of years of real world experience.

For his time spent introducing students to Microsoft’s Visual Basic, Traenk has been awarded the Microsoft Most Valuable Professional (MVP) designation since 2004.   He has also authored  VCP VMware 310 Cert Flash Cards as a late stage exam preparation tool.

In part one of our interview below, Traenk demonstrates the value of business continuity and disaster recovery planning through a look at both the evolution and future of  these solutions:

Evolving Solutions:
You have extensive experience in the fields of disaster recovery and business continuity.  How would you describe the evolution of recovery and continuity solutions since you first entered IT?

John “Traenk” Traenkenschuh:
Circular, the path from older Business Resumption Planning (BRP) options to today’s BRP options seems to have gone in a circle.

In times past, well defined applications were housed on well maintained and highly available hosts.  This design simplified identifying both the critical applications and the business data (and important external datastores) being acted on.  Flash forward a decade or more, and the emergent client-server model has us splattering app bits and pieces all over an increasingly ‘splashed to the four winds’ technical infrastructure.

The UNIX server acts as a client to the z/Series, fetching a copy of Accounts Receivable from some obscure PDS and then acting on it.  The new generation of Accounts Receivable data results are now posted to a Windows 2000 server, where someone’s copy of excel, running on a PC, performs data transformations that a staff assistant posts as authoritative graphs of the organization’s Accounts Receivable status.  And this mix of platforms and informal accesses is driving business decision making!

Indeed, everyone applauds the data without realizing the BRP issues:

•    How can we secure that data (and transformations that occur) across so many network and SAN paths?
•    What constitutes ‘safe storage’ in this ad hoc design?  Are any of the data generations ever reckoned back to the z/Series?
•    Which devices are now promoted to our high-priority computer/application list, those systems that MUST be restored by hour four of our BRP planning?  (And are we really comfortable with important data being manipulated and stored on the Staff Assistant’s laptop, possibly misplaced by absent-minded baggage handlers???)

Now flash forward, again, to 2009.  The right application of virtualization technologies can alleviate many of the harms we thought unsolvable just a few years ago.  We begin centralizing the technical infrastructures into a handful of virtualization hosts.  The mandate to virtualize means the company begins alerting and responding to the ad hoc IT flows that flooded our 90’s networks.

Throw in Desktop virtualization, and even those sore-point endpoints, the thousands of laptops and desktops winking on and off the internal network (so-called ‘Intraverse’), these are now backed up reliably.  (No one is saying goodbye to fat clients with the new scheme either.)

This is the core premise of virtualization technologies, that we might begin returning to required centralized technical and governance structures, structures that allow the organization to meet regulatory requirements, to cut costs, and to begin adopting a more green footprint as hundreds of dedicated computers are folded into a handful.

Evolving Solutions:
What disaster recovery and business continuity solutions do you see emerging in the next 5 years?

John “Traenk” Traenkenschuh:
If the term ‘solutions’ equals IT technologies, I think we start poorly?  BRP has always been a practices and procedures discussion; one implemented through technology to be sure, but one that has never been about technology, per se.

I believe, strongly, that the regulatory pressures and economic costs of today’s IT infrastructures require increased virtualization.  This will begin normalizing the infrastructure, the applications, the data (and access methods—maybe, more on that below), etc.  This will impact BRP in several fundamental ways:
•    Technical infrastructure BRP plans must no longer mirror a fractured infrastructure/Intraverse, one that includes all known and planned flavors of linux, a few Macs in the warehouse, Billy-bob’s mobile phone app, and who-knows-what ancient systems lingering in any one building’s computer center.  We lessen the options and force updates.
•    ‘Stealth’ processing and data results will be identified, making system- and application-prioritization more reliable.  Much like the show, “Cash in the Attic”, virtualization forces us to check into all the dataflows and systems , if we are to achieve our goal.
•    Flows that are difficult to manage may go outside the organization.  Increasingly, internal IT shops are no longer required to host every website nor to code up each and every workgroup-level Word macro.  Some of the processing, lurking in baling wire informal technologies that often run on volunteer hardware, these may need to go elsewhere for support.

I see governance to governmental regulations (and business partner practices) increasing the pressure to change.  If IT organizations cannot get a handle on internal pressures to [mis]manage application design and basic information access, away from longtime informal practices; at some point some-to-all IT services may be moved to Cloud Computing organizations, who will reduce an complex IT equation to a true Software as a Service (SaaS) offering.

At this time, IT organizations are in flux regarding whether to build an internal virtualization infrastructure or whether to vFarm IT Out to a third party.  There are compelling reasons for and against building or sourcing your vFarm.

There may be a middle ground:  ‘enhanced resources’, those you and I call ‘Consultants’.  These will be tasked with mapping legacy organizational practice with externally dictated Best Practices, with the idea that there can be a smooth transition plan to a world-class IT infrastructure.  However, that is an expensive course; and as a former consultant, I know that there are some very insular organizations that will not transition until they must.

In my mind, the Enron, and now banking crises, have made regulatory oversight of most organizations inevitable.  Payment Card Industry (PCI) compliance requires security testing with world-class tools.  Although imperfect, this system anticipates all organizations submitting to third-party security audits.

Please remember that I had seven great years in the Insurance industry, at a premier company.  Insurance works, providing premium sufficiency, because industry members manage to common processes that examine the risk exposures faced by all.  In fact, if I can add one small point, I’m shocked that the world of IT security metrics STILL does not have data-driven risk experience models as sophisticated as those used in the insurance industry.

Ask most insurance pro’s the relative claim value of the loss of mechanic’s finger, and they can arrive at a figure, no matter how obscure the facts.  Ask an IT person the relative value of a hairball analysis application, running at a pet food company with 23% market share, and you will wait, despite so much business data near at hand.

The Center for Internet Security (www.cisecurity.org) has an intriguing system of metrics announced recently.  Mitre.org’s CVSS is some help.  But overall, much remains mysterious, although commodity virtualization infrastructures and service offerings may bring us to a more common understanding of the worth of systems and their data, should a disaster occur.

If readers would like a short list of trackable technologies, those aiding BRP, let me offer this one:

•    Security Information and Event Management (SIEM) – Thanks to responsible vendors publishing long lists of security best practices, (Microsoft and others) and to the work of responsible security think-tanks covering security for multi-vendor environments (Center for Internet Security and SANS (@ www.SANS.org)); many organizations have enabled all the logging we can.  In debug mode at that!  This has made incident response as difficult as finding the proverbial needle in a haystack [of event information].  How can you prioritize BRP responses for the important applications, when you cannot separate the security wheat from the chaff?

•    Virtualization Security (known by many names and techniques) – Now that vFarms are hosting our applications, either on-site or off-site, we need to track what goes on at the virtualization ‘back-plane’.  The reoccurring fear is that a hacker can use a virtualized machine’s (VM) security weaknesses to attack the hypervisor, and then use it as gateway to other VM’s.  Another fear is the ‘rogue’ vFarm administrator who does all manner of bad things, accidently or maliciously.  In this world, the vendors to watch are those virtualization vendors with a long history of security prowess and competent tools AND those security vendors who offer solutions for the virtualization layer and for those VM’s needing their host security tracked and alerted (CA and eTrust and others).  The lack of security API’s in many virtualization packages is a leveling factor; few tools can operate at the backplane layer.  But to be sure, configuration audit and management is still possible.  As security API’s are provided by the vendors, being aligned to a solid security vendor will provide valuable.

•    Risk evaluation tools – As mentioned before, there is a fundamental fuzziness to security evaluation that makes risk mitigation difficult, if not dangerously off-the-mark.  Once regulations and cyber-security governmental appointments begin leveling the playing field, we’ll see new, improved risk models and companion tools that make risk evaluation less subject to personal and professional biases.  Maybe.

•    Green IT Movement – Complementary to BRP is the Green IT Movement.  Whether the computers gain electrical efficiency or we find ourselves growing a more extensive IT intraverse on fewer systems, these factors impact BRP directly.  Uninterruptable Power Supplies may be cut back, either because of fewer/more efficient computers OR because we do not want to proliferate an IT environment full of lead-acid batteries belching hydrogen fumes, possibly spilling sulphuric acid during a disaster.   Computer room temperature control units may be scaled back because of fewer computers, improving BRP focus.  I recently read a toilet paper wrapper that proudly proclaimed that the energy used during production was generated through windmills!  Expect all organizations to be encouraged to offer similar claims to environmental sensitivity—and for reasonable adjustments to be made to our BRP plans.

Part 2 of our interview with discussing Business Continuity & Disaster Recovery with John “Traenk” Traenkenschuh, discussing the factors guiding continuity and disaster recovery planning, and tips for getting a plan started, will publish later this week.

Business Continuity & Disaster Recovery Expert Lawrence J. Webber

Ensuring your business has a solid disaster recovery and business continuity plan in place isn’t just good practice, it can be a valuable sales tool.  With this in mind, we interviewed Lawrence J. Webber for the latest post in our Data Center Leaders interview series.

Along with Michael Wallace, Webber is one of the co-authors of the acclaimed The Disaster Recovery Handbook: A Step-by-Step Plan to Ensure Business Continuity and Protect Vital Operations, Facilities, and Assets.

Below, we discuss the reasons for your business to develop a disaster recovery and business continuity plan, how to get started, and how to use these plans as sales tool in front of prospects:

Evolving Solutions:
What factors play most heavily in developing a continuity plan, for example, government regulations, client contracts, or something more?

Lawrence J. Webber:
Disaster recovery plans are required for government regulations to protect stockholders from a company’s collapse in the face of a disaster (such as loss of a data center, etc.)  Their goal is to quickly restore essential company activities.  Non-essential activities are restored over time.

Business Continuity plans (actions in case of the failure of a significant component) are usually driven by customer requirements.  A reputation as a reliable supplier is valuable sales tool.

Companies providing Just-In-Time materials must have provisions to ensure that they can reliably deliver the expected goods even in the face of a problem.  This could be a need to set up a second assembly line, a second factory or to provide duplicate equipment for all process bottlenecks.

No matter how low your price – no one will buy if you cannot reliably deliver.

Evolving Solutions:
What are the most common misconceptions in regards to what a business continuity plan should or should not entail?

Lawrence J. Webber:
1.     Business continuity plans belong to the Business Continuity Manager.  Business continuity plans actually belong to the process owners, because if the process fails and the plan does not address the problem, it is that process manager who will under the management spotlight.

Since it is their plan, they must ensure it remains up to date and that team members know their roles.

2.    That the Business Continuity Manager (BCM) will “go write us a plan”.  This person coordinates the authoring of plans by others.  The BCM does not fully understand the processes of the Accounting Dept., the materials management group, the engineering team, etc.  Each group must fully participate in the process.  They often imagine the BCM will trot through their offices and magically write a workable plan for each.

Evolving Solutions:
What tips would you offer for a business as it develops continuity plan for the first time?

Lawrence J. Webber:
Don’t feel overwhelmed.  The plan only addresses restoring the critical business functions – perhaps 20% of the total.  Take it in stages.  ID what is most valuable, write a disaster recovery plan, and then write a business continuity plan.

It costs nothing to gather the basic information into one place:

•    Recall list for all personnel (phone numbers, emails, etc.).  Verify quarterly (preferably by calling them. Roster of all vendors, what they supply, and a 24 hour contact number.
•    List of support contracts (contacted via the vendor roster) along with what they support, hours of support, contract number, etc.
•    Build a calendar for when each contract expires
•    Keys to everything, including network cabinets, closets, passwords to servers, etc.
•    Ensure that ALL data residing on data center storage devices is backed up and then promptly moved off site to a secure storage area.  Verify that these back ups work, know who can recall the data and how to do it.
•    Identify critical IT systems, and the primary and secondary support person for each.
•    Ensure each person is on the recall list
•    Identify the critical components for each (servers, peripherals, etc.)
•    Ensure these items are covered by vendor support agreements

Evolving Solutions:
Susan Snedaker,  Principal Consultant with VirtualTeam and author of Business Continuity & Disaster Recovery Planning For IT Professionals, identified as the three biggest mistakes when developing a continuity plan as “Not Creating A Plan,” “Not Getting Executive Buy-In,” and “Not Getting The Right People In The Room.”  What would you add to this list?

Lawrence J. Webber:
False confidence that once a plan is written, you are safe.  It must be regularly tested (perhaps quarterly) so that everyone knows their roles and that the plan reflects the current processes.  A plan sitting on a shelf is a snapshot in time.

Processes change, so do process tasks and staffing.  The document does not magically change by itself, and often no one bothers to inform the Business Continuity Manager.

Evolving Solutions:
How would you measure the chance of a newly launched company’s success, with or without, a disaster recovery or business continuity plan?

Lawrence J. Webber:
A disaster plan and a business continuity plan are only called into action when something goes wrong.  A new company with potentially excess capacity can disguise a disruption from a customer.  However, a well run company, tightly staffed, cannot disguise a disruption.  At best, they are tempting fate.

Evolving Solutions:
Wild Card: Anything else you’d like to add?

Lawrence J. Webber:
Disaster recovery is all cost.   Like insurance, you pay and pay but usually never need it (ie no disasters strike).

Business Continuity planning provides payback in resilient processes which result in more reliable cost estimates and product/service delivery. Green and Lean initiatives (such as virtualizing servers) also shortens recovery time.

Vote 2 Evolve your company’s IT by completing a brief IT assessment survey.  Vote 2 Evolve your Minnesota Wild by selecting the minor league Houston Aeros player you think will be called up first.

When you do, Evolving Solutions  will register you for a chance to win an officially licensed Marian Gaborik jersey and follow-up with details on IT solutions designed to reduce your data center costs and improve your business continuity.

Complete our survey today to qualify instantly for reduced IT costs, improved business continuity and an officially licensed Marian Gaborik jersey.

Vote 2 Evolve by June 15th to qualify.

Good luck!

Business Continuity Expert Susan Snedaker

Data center professionals are in a unique position in today’s marketplace.  It is data center professionals who develop  in-demand strategies designed to do the most work with the fewest resources, whether it’s minimizing costs by virtualizing physical servers or creating sound data recovery plans ensuring companies recovers from natural or man-made disasters.

Evolving Solutions is proud to launch our blog’s newest feature, “Data Center Leaders.”  Evolving Solutions will interview data center leaders for their thoughts regarding topics ranging from server virtualization to business continuity with everything in between.

First up, is our interview with Susan Snedaker.  Principal Consultant with VirtualTeam, and author of Business Continuity & Disaster Recovery Planning For IT Professionals, Susan is an accomplished consultant, speaker and author. Equally versed in business and technology, Susan specializes in defining successful business models that increase profitability, reduce turnover and define a clear vision for future success.  Susan’s insight can also be found at her blog, Starting Up, Starting Over – Business Fundamentals.

Below, Evolving Solutions discusses business continuity and disaster recovery planning with Susan:

Evolving Solutions:
What factors play most heavily in developing a business continuity plan, for example, government regulations, client contracts, etc?

Susan Snedaker:
The factors that should be considered vary depending on the nature and size of the business. A large hospital will have to make very different decisions than a mid-sized optical manufacturing company or a small online retailer. The key considerations are tiered in this order:
1.    Government, legal or regulatory requirements
2.    Industry requirements
3.    Corporate requirements

For example, the hospital must comply with FDA requirements, HIPAA requirements and a whole host of other legal and regulatory requirements in the daily course of business. These should be primary considerations for any BC/DR plan. The manufacturing firm may have to comply with OSHA or EPA standards during the course of business. The online retailer may have few, if any, regulations governing their business activities.

Industry requirements may include adherence to certain standards. For example, in manufacturing, there may not be a governmental regulation of the product but there may be stringent industry requirements for precision, purity, etc.  Again, during the normal course of business, these things are typically addressed  in standard operating procedures and should be included in the BC/DR plan.

Corporate requirements include critical business applications, data and processes along with vendor and client contractual commitments. Using the same examples, the hospital must meet the needs of a variety of stakeholders (with respect to BC/DR) including patients, the community, medical supply providers, physicians, nurses and other health care providers. Each of these groups has specific needs and requirements that all focus on patient care and these form the foundation of the BC/DR requirements.

The manufacturing environment may focus on meeting contractual obligations with regard to just in time inventory management, logistics or sourcing to name a few. The online retailer may have contractual obligations with vendors for purchase levels or frequency of purchases or they may have specific obligations with respect to turning around customer orders.

Most companies these days are using a variety of technology solutions and each of these must be assessed as to their criticality in the functioning of the business.  Companies also have to address the interdependencies of systems and the order in which they would preserve and restore systems. Having assessed the regulatory environment, the firm can better assess which business data and functions should be considered highest priority.

In a hospital environment, life support systems  and those regulated via HIPAA or the FDA would be at the very top of the list while the gift shop inventory system may be at the very bottom of the list, for example. The manufacturing firm would include any systems used to manufacture product at the top and perhaps standard office systems (word processing, etc.) at the bottom of the list. The online retailer would probably consider their web-based shopping cart system to be their top priority followed by the inventory system then other internal systems.

If you approach the creation of a business continuity/disaster recovery plan from the top down, you’ll likely take the most important factors into consideration first.

Evolving Solutions:
What are the three biggest mistakes companies make when developing continuity & disaster recovery plans, and how can they be avoided?

Susan Snedaker:
Mistake #1 – Not Creating A Plan

The biggest mistake companies tend to make is to not create a plan at all. If you ask a room full of IT professionals how many of them have backups of key data on their home computers, you’re likely to find that perhaps 10-20 percent of actually do backups at home.

Clearly, IT and other business professional know they should have a plan but they rarely do. The biggest roadblock to creating a plan is often the seeming enormity of the task. Large companies  may choose to contract with third party providers to assist them through the process rather than re-invent the wheel. There are proven methodologies for assessing the company’s business continuity and disaster recovery needs . Stepping through  a defined process on an enterprise-wide basis yields a more reliable plan than an ad hoc approach.

Mistake #2 – Not Getting Executive Buy In

If you don’t have executive support for your business continuity and disaster recovery process, you’re not likely to make much progress. Creating a workable business continuity and disaster recovery plan can be time-consuming and (depending on your company and industry) expensive. You need to have executive support to help you get all the needed players to the table across the entire company.  You may also need to educate your executives about the cost of NOT creating a workable plan.

Mistake #3 – Not Getting The Right People In The Room

If you don’t have executive support, you may have trouble getting the right people to put in the requisite time and effort to create a viable business continuity and disaster recovery plan. Even with executive support, some companies miss their target because they create the plan in an information vacuum then try to roll it out to the organization.

Instead, each key department should have a representative weigh in during the creation of the plan to ensure it meets the entire organization’s needs. It often falls on the IT group to create the business continuity and disaster recovery plan, but in a hospital , manufacturing  or other complex environment, it’s not likely that the IT staff will have enough knowledge about daily operations to ensure that the plan is realistic.

Evolving Solutions:
What tips would you offer for a business as it develops a business continuity & disaster recovery plan for the first time?

Susan Snedaker:
Start with your data. What is your most critical data? Where and how is it stored? Create a viable plan for backing up and recovering your electronic data in the event of catastrophic loss. If your server room imploded, what would you do?

Do you know what kind of equipment you’re running, where you could purchase duplicate equipment, how you could restore your data to new equipment in an alternate location?  Do you have copies of operating systems, patches, configuration and passwords off-site in a secure (but accessible) location? Many companies don’t even cover the bases with adequate backup and restore capabilities and that’s the best place to start for all companies. Once you’ve secured your data, you can then enlarge the scope of your business continuity and disaster recovery plan.

Creating a business continuity and disaster recovery plan, especially for small and medium-sized businesses, is likely to be an iterative process where data is secured then physical assets then business processes. The bottom line: Keep it simple but create a basic plan.

For example, the online retailer may have a very simple business continuity and disaster recovery plan. They’ve ensured (contractually) that their web hosting company has a disaster recovery plan for web services. Their inventory database and financial system (QuickBooks (R)  most likely) is backed up using a real-time incremental backup service that backs data up to a secure Internet site during low usage times. Inventory would have to be replaced if the building was damaged, but with a new location and a couple of computers, the online retailer’s back in business.

Clearly, that’s the simple version but it shows that with just a bit of planning the basics can be covered. The online retailer can then go back through their plan once they get these pieces in place and begin planning for other potential problems such as the building being damaged or transportation to their facility being interrupted. The manufacturing company and hospital will have a much more complex plan, but it uses the same process and starts with securing critical data.

Evolving Solutions:
Your book, Business Continuity and Disaster Recovery Planning for IT Professionals takes the reader step by step through the process of developing their own continuity and disaster recovery plans.  Taking away the regulations of specific industries, do you feel the general process of creating a plan is able to be duplicated for most companies?

Susan Snedaker:
Yes, the process for business continuity and disaster recovery planning can be duplicated, which is why there are service providers out there who can be hired to assist in the process, regardless of industry. However, as you’ve seen, the details vary greatly from company to company.

The basics really start with protecting key data. Don’t fall into the trap of thinking it’s too big a job to complete so it never starts. Break it into manageable pieces and protect your data. Be clear about what is and is not included in the project so your CEO or CIO doesn’t incorrectly assume you have a full, robust and complete business continuity and disaster recovery plan if all you have is a solid data protection plan.

Evolving Solutions:
In 2006, CIO Magazine reported that many existing business continuity plans would likely fail in the instance of a global pandemic, as most plans were created to only take into disruptions caused by geographical disasters.  Two years later, do you feel this is still the case?

Susan Snedaker:
Most companies would probably not be ready for a pandemic, even now, but I’m not sure any government on the planet is really ready for a pandemic either. It’s an enormous scenario to consider.

However, I think companies are more aware of the potential for a pandemic and as a result, they’re beginning to consider these possibilities. In an economic downturn, companies scale back on non-essential costs and that often includes business continuity and disaster recovery planning. So, they’re most likely concentrating their efforts on ensuring critical data can be recovered and core business functions remain in tact and anything outside that scaled down scope has probably been cut loose. I would say most companies are prepared only to the extent the company’s primary business continuity and disaster recovery plan is also applicable in a pandemic.

Evolving Solutions:
Wild Card: Anything else you’d like to add?

Susan Snedaker:
1.    Some interesting statistics your readers might find of interest. The most common disaster companies face is fire.

2.    The chances of a company staying in business after a “disaster event” (fire, flood, etc.) are directly correlated to how quickly they come back up after the event. The longer you’re down, the less likely you are to remain in business long-term.

3.    If your firm is scaling back on IT assets or investments in this economic climate, there’s a good chance it’s canceling or closing out disaster recovery contracts to save money. Be sure you review your plan and your contracts. Scale back if you need to, but update your plan accordingly and realize that you are exposing your business to additional risk. Though you may have to scale back, if you review your business continuity and disaster recovery plan you may find ways to save money on existing contracts and services in a soft economy rather than scrapping your plan altogether. The key is to make thoughtful decisions rather than yanking the plug on a plan and hoping for the best.