Virtualization Expert Victor Avelar

What is the bigger incentive to run an energy efficient data center: saving some money or saving the earth?    Can a data center manager consider both to be equally important or is this line of thinking inherently dangerous?  And where do technologies like server and storage virtualization fit into the mix?

To answer these questions and more, our Data Center Leaders Interview Series turned to Senior Research Analyst from APC Victor Avelar, fresh off his webinar with Aaron Goldberg, Unleashing Virtualization’s Full Energy Savings:

Evolving Solutions:
What tips would you offer for business seeking to reduce data center costs?

Victor Avelar:
In general, the older the data center, the more energy-saving improvements that can be made.

Cooling system improvements are always a good place to start.  If hot and cold aisle containment hasn’t been implemented yet, I would start planning this, so that all future refreshes result in some kind of rack layout relocations that bring the data center closer to 100% hot / cold aisle.

Hot / cold aisle configuration will increase the effectiveness of air distribution and could allow one or more of the air conditioners to be turned off.  Same thing with installing blanking panels in all the empty U spaces of the racks.

Another easy way to save energy costs is to unplug servers that are no longer needed yet are still plugged in.  If there are any windows in the data room / center, cover them up with some insulation to reduce the solar heat gain that the air conditioners must remove.

Evolving Solutions:
Are there inherent dangers in trying to make your data center too cost efficient?

Victor Avelar:
In short the answer is yes.  You can place your critical operations at risk if you don’t understand how an ‘improvement’ impacts other systems.

For example, I could really save energy by turning off some air conditioners but if I’m not careful about monitoring temperature, a few servers may turn off due to thermal shutdown.

However, there’s nothing wrong with going all out to make your data center more cost efficient, the key is to know when you begin to hit the point of diminishing returns.

Evolving Solutions:
The webinar you co-presented with Aaron Goldberg, Unleashing Virtualization’s Full Energy Savings, highlights tactics like server virtualization that will help attendees maximize data center efficiency.  Are solutions like server virtualization geared more towards large data centers, small data centers, or can these solutions be scaled for any size company?

Victor Avelar:
I believe the largest % savings from virtualization go to companies that have hundreds of servers.

Host servers running 40 to 60 virtual machines is very possible and very efficient.  Large companies also have a better chance of rightsizing their power and cooling systems which can reduce the electric bill even further.

This isn’t to say that companies with 50 servers couldn’t benefit, but the savings percent tends to be lower.

Evolving Solutions:
According to survey results published by The Data Center Users Group and reported by SearchDataCenter.com, 47% of data center managers list energy efficiency as their second highest concern.  In a similar survey published four years ago, energy efficiency did not even appear in the top three.  Outside of the desire to save costs, what do you feel has driven the concern over data center energy efficiency so high?

Victor Avelar:
I believe there are a few different reasons for this.  One is definitely psychology.  If my peers are actively pursuing energy efficiency, you can bet that I will follow the herd, since I don’t want to appear clueless.

On a similar vein, the “green movement” has taken hold and companies want to be seen as part of the solution when it comes to saving the earth.  The risk of backlash is too great if they don’t.

Furthermore, if many people are pursuing it, chances are there is some real value there.  Which brings up reason #2; savings.  Done right, energy efficiency can definitely lead to real savings on the electrical bill.  Reason #3 is government regulations.

It’s just a matter of time before the data center industry is acted upon by energy regulations through carbon tax, Energy Star incentives, penalties, and other ways to reduce energy consumption.  Some electric utilities are already providing incentives for companies to reduce power consumption through rebates for buying certain types of IT equipment.

Evolving Solutions:
What products or solutions do you envision contributing to data center efficiency 5 years down the road?

Victor Avelar:
By far the biggest positive impact on energy savings will come from cooling system improvements.  If you look at the efficiency of the power train (i.e. from service entrance to the rack power strips) we have solutions today that result in a power train efficiency of greater than 86%.  The “cooling train” is far less efficient and is fertile ground for improvements in the next five years.

Specifically we’ll see solutions related to free-cooling through air-side and water-side economization.  Of course, the energy savings of these solutions will depend heavily on where in the world you locate your data center.  Other improvements in cooling systems will come from “smart” data center management which makes certain changes to devices based a holistic understanding of the entire ecosystem.

Today, a “good” change made to one device may cause the net power consumption to increase.  The classic example is when one air conditioner is cooling and humidifying while another one is dehumidifying.

Another solution that we’ll see is data center management that not only monitors and controls the white space but also the electrical and mechanical systems.  These management systems will have a holistic view of the entire ecosystem and will be able to alert when the data center efficiency is lower than it should be.

Evolving Solutions:
Wild Card:  Anything else you would like to add?

Victor Avelar:
There is so much information out there today and for the most part it’s pretty accurate.  I encourage all data center operators to read what’s out there so they are at least familiar with various solutions and how they decrease power consumption.

Antivirus Researcher & Disaster Recovery Expert Peter Szor

Truly one of the most insidious of the man-made disasters threatening a company’s data center are ever present, ever-evolving, computer viruses.

On the front lines helping to backup and protect data centers from these viruses is Peter Szor, the subject of today’s Data Center Leaders Interview.

Szor is a computer antivirus and security researcher with 20 years of experience building antivirus and security solutions. He is a distinguished engineer at Symantec Corporation, holds over 30 issued computer security patents and is the author of the best selling technical book The Art Of Computer Virus Research and Defense(1).

Below, we discuss the history and rapid evolution of computer viruses, answer why companies should think broader than antivirus protection when creating disaster recovery plans, and overlook the damage caused by a few of the most malicious viruses Szor has ever seen:

Evolving Solutions:
In your experience, how many companies understand the importance of creating disaster recovery plans that take cyber-threats, like computer viruses, into account?

Peter Szor:
Large companies with critical infrastructures always understood the risk of computer malware.

Yet, while antivirus solutions remains their number one security choice, relatively few companies are focusing on other important aspects of computer security that are also critical to them, such as making sure data is kept secure, yet available at all times.

Companies know that well maintained computer antivirus solutions from trusted, dedicated security vendors will vital for their security, but I strongly believe that in the very near future companies will move to the next stage of understanding and put more focus on disaster recovery plans.

Companies need to understand that mitigation of risks is increasingly important, especially in environments where services are provided to users where any interruptions can quickly lead to negative business impact.

It is amazing how much the data needs of companies have changed over the years.  Even in homes, people store far more data than ever before as everything is increasingly becoming digital. Both companies and home are at risk of losing information, which needs to be protected and kept secure from viruses and protected by data backups.

Having spent almost 20 years developing computer antivirus and security software, I strongly feel, that customers understand security risks much more today than ever before. Fast spreading computer worms such as CodeRed, Nimda, Blaster and Conflicker all point to the same underlying issues in our networks that we share. Computer networks, operating systems and applications will remain vulnerable and exploitable, and disaster recovery plans are more important than ever. Data backup and storage management are critical.

Evolving Solutions:
What tips would you offer businesses to protect against computer viruses that seem to evolve just as quickly as software designed to prevent infection?

Peter Szor:
We often see that companies do not manage their infrastructure enough. Vulnerabilities are not always managed by deploying security updates at all end points. Companies do not always follow closely enough what software their users run, what they can do, and what attacks they might bring to the corporate network when doing so.

Operating systems are often old, as are the applications on them, meaning they often have vulnerabilities unpatched. Most users run their system as Administrator. Today, what we find is that the majority of attacks are getting in via downloads when users browse the web. Securing the browser is critical. What the user can browse has a huge impact on the internal security of the network, especially so, when the code enjoys Administrator privilege to get installed right away.

Of course, I would also recommend user education. I do not give up my hopes in this regard. My family dedicated their life to education: math, physics, history and music, you name it. I tried to contribute to the field of computer security myself and hope that computer professionals understand attacks and defenses better from my work. I am happy to see that computer security is becoming a science and that people can graduate by receiving degrees in the subject.

I strongly recommend companies to hire security professionals with first-hand experience in security. The degree is one thing and the experience is the other. The more people understand computer security, the better it will get for the company. Yet, if there is no expertise in house, security consulting should be used.

Computer security is evolving all the time with the new attacks. It is exciting to see how much antivirus has evolved over the last few years as the threat space grow to over 4 million. To me, this is an amazing expansion of the threat space.

For the first 10 years, we have witnessed just about 10,000 distinct computer virus variants and we all believed it was already overwhelming. We got the rest of the malware space during the last decade which clearly shows an exponential curve. The malware universe expansion is clearly rapidly accelerating.

Keeping your version of Antivirus engines and products up to date is important beside the definition data provided. There are true inventions in AV software today, such as advanced heuristics, software behavior management and “cloud” based reputation systems, which will all shape client protection during the next decade. These inventions come with new products which need to be deployed time to time to be sure that computer security can evolve with the new threats.

Evolving Solutions:
Your book, The Art of Computer Virus Research and Defense, is a report from behind the scenes in anti-virus research.  Apart from what is published in your book, what is the one piece of data most important for businesses to know regarding how a virus could affect their data centers?

Peter Szor:
There are many attack types, which – fortunately – have not developed to their full potential.

During the last few years, exploitation was the main focus of attackers. What we see today is that web browsing brings more and more attacks to the end points, and so, we made our defense stronger against such malware attacks.

Computer viruses can cause devastation, especially the fast spreading worms that open up the network to the remote control of the attackers. When confidential information is leaked, there is always a problem, which goes way beyond the recovery of the attacks itself on the internal network because it affects the reputation of the company who leaked the data. Therefore the protection against information leakage is increasingly important aspect of security today.

I already mentioned that so called “cloud based” security solutions will shape the security landscape during the upcoming years. Targeted, unique attacks are exploiting end points every second. It is not unlikely that we will see more than 10 million malware variants during 2010.

People who are behind these attacks operate as businesses, and make a lot of money, which they can reinvest to improve attacks. Unfortunately, this process accelerates the evolution of malware a lot.

If you think about it, attackers already use cloud computing, when they harvest bot networks for their use, such as spam delivery. Next, I think, they will increasingly use real cloud computing systems, since they can effort to borrow as many virtual machines as they want relatively cheaply, and they can certainly effort to pay for them as needed.

Modern attacks require revolutionary security software to address them, and this is precisely what we are working on.

Evolving Solutions:
How has Symantec learned to anticipate computer virus evolutions and develop software to combat these viruses accordingly?

Peter Szor:
Symantec was the pioneer of fast antivirus updates.

We realized that instantaneous pulsing update processes were important and eventually we invented the idea of providing a service directly to clients querying a central database. This provides the most up to date security protection.

Software reputation services will be a strong pillar of our computer security. We built a large software reputation database for the last few years and are getting ready to use it. With this, Symantec will help users to avoid software, which is rarely used, as most Trojan programs are very rare with few victims each.

We fight back against server side polymorphism – the effect behind the quick millions of malware increases – by realizing that users typically want to run software that many people also use. When you choose a restaurant, you want to be sure that is grade A, and have good food, and you know that if you see that the place is always packed. When you see a grade B restaurant with a few people inside, you want to avoid it, because you risk that you get sick when eating there. Similarly, if you are among the first to run a program that nobody has ever ran, you better not to take the risk. Such a policy will help protection tremendously in the future, and possibly, it is the greatest extension of the art of computer protection since my book was published a few years ago.

We understand, that our protection against malware attacks such as self-replicating viruses and worms is very strong, and thus, traditional techniques help our customers to fight back against them. We made sure during the last decade that our software provides solid protection against even the most sophisticated polymorphic and metamorphic virus attacks. We demonstrated in leading antivirus tests that we are unmatched when doing so. We made our protection against malware attacks much stronger during the last 12 months, while improving the performance of our software at the same time.

Evolving Solutions:
What is the most despicable computer virus you have ever witnessed, and without naming company names, what level of damage did you see it cause?

Peter Szor:
During the years, I have seen successful attacks, which deleted data that could not be restored since data backups were typically not available. We have even witnessed PC’s being destroyed by overwriting the content of their Flash-BIOS, as the CIH virus did, that made the motherboard of the attacked system useless.

You could not possibly prevent using add on software- the Flash-BIOS- to be overwritten, since the “metal” could be directly accessed via PORT commands with no way of interruption, once the malicious code ran on the system. This is a basic design flaw of modern computer architectures.

Instead, the actual viruses and Trojans had to be detected at the first place before they could run. Antivirus software was key to detecting these attacks and will surely remain the wheel of computer security in the future.

Back in 1995, I was certain that Windows systems would be the new target of attackers and expected to see computer worms on the platform. First, I witnessed the Happy99 worm, released a decade ago, which demonstrated the main problem infecting systems world wild.  Then, attackers finally turned towards the use of exploits.

CodeRed and Nimda worms would show how quickly attacks could spread on the Internet when exploiting remote vulnerabilities.  When I traveled to Europe in September of 2001 to visit the Virus Bulletin conference in Prague. I recall, the cab driver asked me, what kind of business I did, and I proudly said:

“I am a computer antivirus researcher.”

He quickly went on to say:

“Have you heard of Nimda, Admin backwards? It is all over the radio!!”

When he noticed that I had no idea what he talked about – I just landed in Prague a few minutes earlier, and the worm was actually released while I was in the air – he went on saying laughingly:

“What kind of security researcher are you?”

At that very moment, we both realized that the security world had dramatically changed. Then all other researchers at the conference talked to me about Nimda one by one. They all knew I was painstakingly analyzing every single variant of Win32 malware, carefully cataloging them, and giving them their names.  Then, the sudden explosion of these threats just happened, seemingly one day to the next.

The Conflicker worm recently demonstrated that essentially the same vulnerabilities are still with us. As a matter of fact, Conflicker uses some modules that were built years ago by the 29A virus-writing group, which is no longer, but their legacy is still with us.

Certainly, there is more to do to improve protection at both sides of the spectrum: at security vendors as well as at the end points by the companies themselves. We are working very hard to improve security for our users who can be rest assured that we have never, ever been more focused on delivering the best protection in the industry.

(1) The Art Of Computer Virus Research and Defense, published by Addison Wesley 2005.

Grow A Greener Data Center by IT Architect Douglas Alger From Cisco

“Green IT is ultimately about resource efficiency – maximizing what you have and optimizing how it is consumed.  Beyond the admirable environmental benefits from being green, such optimization lets you accomplish more and spend less.”

This is just a sampling of the insight you’ll find below as our Data Center Leaders Interview Series details necessary tactics and quantifiable benefits for taking your data center green.  Helping us is Douglas Alger, IT Architect for Physical Infrastructure from Cisco and author of the book Grow A Greener Data Center.

For anyone who still feels that Green IT is simply an altruistic business fad nearing an end, rather than a significant strategy for saving capital and optimizing power usage, this interview is for you:

Evolving Solutions:
What tips would you offer for businesses prior to launching a Green IT initiative?

Douglas Alger:
Specific to  the Data Center space, I recommend having tools in place to monitor your Data Center resources.

Being able to track power consumption down to the cabinet level, temperature conditions, and hardware utilization is invaluable for developing and implementing Data Center green initiatives.

Monitoring tools can help uncover inefficiencies such as short-cycling of an air handler or which servers in your Data Center are consuming a disproportionate amount of power.  They also can help you prioritize potential improvements, calculate the return on investment, and – after implementation – accurately track efficiency gains.  Without them are you left to guess at the impact of the improvements you are making.

Evolving Solutions:
How would you recommend selling the idea of green IT to upper management who see it as little more than a fad?

Douglas Alger:
I think it’s important to emphasize that green IT is ultimately about resource efficiency – maximizing what you have and optimizing how it is consumed.  Beyond the admirable environmental benefits from being green, such optimization lets you accomplish more and spend less, conditions that are always going to be of interest to a company.

For Data Centers, being green saves money by reducing energy consumption (the largest operational expense of a Data Center) and cutting down on the use of consumable items such as patch cords.  Being green also extends the lifespan of your facility (deferring future construction costs), provides more flexibility to accommodate future technologies, and positions your company well in the event that environmental regulations around energy-efficiency or carbon emissions are enacted in the future.

Evolving Solutions:
With business movements that emerge quickly, such as Green, it is not uncommon for mistakes to be made due to rapid deployment.   What implementation mistakes have you witnessed, in regards to Green IT initiatives?

Douglas Alger:
Companies sometimes launch green initiatives without developing an overall strategy.  That’s fine for the short term and individual green projects can definitely be successful, but as more uncoordinated green activities are initiated you can end up with redundant efforts and other inefficiencies, especially at large companies.

Any green initiative that you can think of – recycling programs, promoting alternate transportation, server virtualization, etc. – are bound to accomplish more if they’re implemented as part an organized program with defined goals.

Evolving Solutions:
Your book, “Grow A Greener Data Center,” walks companies through a bottom-up approach to building a green data center, beginning with physical construction.  For companies that do not have the option to physically build a new data center, what do you recommend as an ideal starting point?

Douglas Alger:
There are a lot of green improvements that can be made to existing server environments that are very low cost, paying for themselves many times over, and easy to implement.  One simple step, for example, is to install timers and motion sensors on your Data Center lighting, causing non-emergency lights to turn off whenever the room is unoccupied.  (This can be done for other building spaces that are unoccupied for extended periods of time as well, such as conference rooms and bathrooms.)

If your Data Center’s power and cabling infrastructure are routed under a raised floor, seal any excess gaps at the floor tile openings where patch cords and power cables enter the plenum space.   This will improve the performance of your cooling system, reducing your energy consumption.

Yet another green improvement that can be implemented unobtrusively is to make energy-efficiency a key purchasing criteria for hardware that goes in your Data Center.  Newer, more energy efficient devices can be introduced as part of your company’s normal refresh cycle for hardware.  This can lead to significant savings over time.  When you factor in a server’s cooling needs and the conversion losses that occur along a Data Center’s power delivery chain, your energy savings will ultimately be nearly three times the number of watts you conserve at the hardware level.

Evolving Solutions:
Toby Velte, Global Technology Strategies with Microsoft, describes how he helps to ensure Green IT initiatives are funded by always relating projects to the pressures of capitalism, rather than the pressures of altruism.  Do you find this to be true across the board, or have you seen some firms consider start to implement green IT purely from a sense of corporate responsibility?

Douglas Alger:
I have seen businesses undertake green activities because they want to do the right thing, but I agree that if you want to truly entrench green initiatives within your company you need to demonstrate their business value.  When budgets get tight, upper management is more likely to cut a “feel good” program than one known to contribute in a proven way to the company’s success.

Evolving Solutions:
Wild Card:  Anything else you would like to add?

Douglas Alger:
Some people don’t immediately associate Data Centers with opportunities to be green.  With the tremendous consumption that occurs in these facilities, though – often 20 to 40 times the energy usage of traditional office space – there are ample opportunities to be more efficient and thereby save energy, save money, and reduce your carbon footprint.

Storage Consolidation Expert Rajeev Bhardwaj

Behind labor, energy costs are emerging as the second highest operating cost in most data centers worldwide. Operating in an economy still in the early stages of recovery, companies must explore all options available for reducing data center energy costs.

Today’s entry in our Data Center Leaders Interview Series sets out to help companies make the critical IT decisions that will lower costs, including those associated with storage consolidation, without hindering performance.

For help, we turn to Rajeev Bhardwaj, Senior Director, Data Center Switching Technology Group, Cisco:

Evolving Solutions:
What do you recommend as a first step in the development of a cost efficient storage consolidation or networking plan?

Rajeev Bhardwaj:
End-to-end virtualization will consolidate the datacenter and will deliver operational efficiencies from the perspective of being cost-effective and applications being easier to manage, provision, and deploy. End-to-end virtualization in the datacenter includes server virtualization, server I/O virtualization, Storage Area Network (SAN) virtualization, and Storage virtualization.

End-to-end virtualization should begin with SAN virtualization as the SAN network interconnects all the storage and server elements in the datacenter and a consolidated SAN can serve as a platform for further virtualization of servers, server I/Os, and storage.

Customers typically deploy silo’ed infrastructure with multiple SAN islands based on departmental, application, and performance needs. The consolidated SAN must maintain the logical separation between the SAN Islands from both data path and control plane perspectives while consolidating them on a single, physical infrastructure: Virtual SAN (VSAN) allows to easily achieve that virtualization abstraction.

Consolidating multiple SAN islands in a virtualized SAN allows to significantly reducing the number of infrastructure assets required, being now shared and dynamically allocated based on actual needs.

This approach optimizes assets’ utilization and dramatically reduces deployment, management, as well as power, space and cooling costs. A consolidated SAN has the additional advantage that intelligence can now be deployed in the SAN to enable both server virtualization and storage virtualization.

Evolving Solutions:
What are the inherent dangers in trying to make your storage consolidation plan too cost efficient?

Rajeev Bhardwaj:
It is very important to consider the underlying architecture of each device required to deliver the consolidated SAN. If the SAN switch architecture is not robust and delivers unpredictable performance and latency, the consolidated SAN becomes a central bottleneck affecting all applications and all departments.

One should consider questions such as – how does the switch architecture handle congestion, does it exhibit consistent latency and performance, does it provide enough buffer-to-buffer credits per port to handle different applications and distances, does it provide high availability and in particular for links that carry a lot of bandwidth between switches?

Evolving Solutions:
Green IT is seen by many as a solution to upgrading existing data centers to be more cost efficient.  Do you feel green IT practices can significantly impact the cost of storage consolidation?

Rajeev Bhardwaj:
A piecemeal approach of deciding each IT component based on its power consumption can work fine for marginally reducing power as a first step, but can actually be counter-productive overall if a holistic view of the data center power is not considered.

Solutions that at the chassis level seem to provide slightly better power consumption, have proven to increase the overall power cost of the datacenter: poor network support of server virtualization constraints the consolidation of servers, consuming over 75% of the overall datacenter power budget; limited networking capabilities and unpredictable performance require to increase network size as soon as new applications are deployed in the datacenter, transforming the initial minimal saving in a major cost.

Green IT can reduce cost and power usage while increasing operational efficiency in the data center if achieving end-to-end virtualization is maintained as a goal. Deploying the right networking infrastructure can provide security, performance, Quality of Service (QOS), and ease of management required to enable reductions in the size of the server farm, the number of LAN and SAN switches, and the number of storage arrays.

Evolving Solutions:
In a recent interview, author and StorageIO found Greg Schulz helped dispel the myth that storage networking and consolidation are business expenses that only apply to large enterprises.  In your experience, are smaller businesses aware of the need for storage consolidation and networking plans?

Rajeev Bhardwaj:
In these tough economic times, smaller businesses are looking to reduce both capital and operational expenditure. Smaller businesses understand that times are tough but are also looking to position themselves for the turn-around in the economy and so are not willing to sacrifice on scalability.

IT is increasingly considered as a means to increase the company efficiency and competitiveness by providing real-time data to any sales person, partner, or executive anywhere at any time while maintaining security and keeping costs low.

Smaller businesses are very much interested in buying and maintaining fewer servers, fewer switches, and getting more mileage out of their existing storage arrays. Thus we see smaller companies collapsing their multiple fabric switches into a larger director to gain scalability and high availability while reducing management expenses.
Similarly, we have also seen keen interest in our Fibre Channel over Ethernet (FCoE) portfolio from smaller business looking to consolidate their LAN and SAN expertise and infrastructures.

Evolving Solutions:
Omar Sultan, your colleague from Cisco, envisioned matured virtualization, greater use of automation and the evolution of cloud computing as solutions that would lower data center costs five years from now.  What would you add to this list?

Rajeev Bhardwaj:
Most organizations have separate LAN and SAN departments managing separate networks today. Five years from now I see this distinction being blurred and administrators who are aware of both LAN and SAN environments running a common network based on common switching components.

I see this common, cost-efficient network providing the under-pinning for higher data center virtualization, automation, and ultimately evolution of highly scalable, secure, and highly available private and public cloud infrastructures.

Evolving Solutions:
Wild Card:  Anything else you would like to add?

Rajeev Bhardwaj:
Energy costs are emerging as the second highest operating cost (behind labor) in most of the data centers worldwide. While networks consume a very small percentage of the overall data center power, deployment of intelligent networks can drastically reduce end-to-end power consumption and management costs while reducing provisioning time for new applications and achieving greater operational efficiencies in the data center.

Our Discussion With John “Traenk” Traenkenschuh Continues

From natural disasters such as tornadoes, to (recently declared) pandemics like swine flu, our world practically demands companies protect data with business continuity and disaster recovery plans.  And yet, some do not.

In part 1 of our Data Center Leaders Interview with IT veteran John “Traenk” Traenkenschuh, we discuss how business continuity and disaster recovery plans have evolved and how they may look years from now.

In Part 2 below, John leverages real world insight, including that  gained from his experience as a ‘Wave One’ responder in the wake of  Hurricane Andrew, to discuss the  factors that should compel all companies to develop strong continuity and disaster recovery plans:

Evolving Solutions:
Dan Blacharski of IT World ran a poll asking readers to share the disaster recovery plans they have in place.  As of this writing, 31% of respondents answered ‘backup software with off-site storage.’  However, almost as high a percentage of respondents had no plan.  What do you find most surprising about these results?
(Editor’s Note:  Poll numbers have since updated to 40% ‘backup software with off-site storage, 28% with no plan.)

John “Traenk” Traenkenschuh:
I’m surprised that organizations are discussing, publically, those problems that they are screening privately.

How many organizations must admit that not only are their backup media stolen, the same media are unencrypted as well?  Too many organizations have no BRP plan and associate BRP with system back up only, whether a prioritized list of core applications exists or not.

These organizations are most vulnerable to a massive disaster that breaks operations for hours, if not days.  Too many organizations have no identified BRP coordinator.  Most have never had a practice outage.  Most have no NOC (Network Operations Center) documentation to guide a BRP event.  Just as bad, too many have documentation done without the help of the technical staff.  These plans are full of ‘smoke-and-mirror’ assumptions such as ‘IP just plain happens here’.

Whether swine flu, bankruptcy, regulations, or natural disasters like tornadas (what happens to a tornado that swings though my part of the country); there have never been a more compelling set of factors requiring true BRP expertise.
Sadly, never have our most prominent organizations been least prepared to cope with today’s events.

Don’t believe me?  Read the news.  Find out how much critically important data is kept on laptops.  Experience the horror as a backup media theft has a ripple effect that has hundreds of thousands of customers given public notice that an organization is negligent at best.  A good Business Resumption Plan anticipates these exposures and helps the organization take steps to lessen impacts.

Most concerning are those stories that reveal significant parts of the infrastructure are open to attack.  Surprised by the results of the poll?  Not at all.  Hopeful those will change?  Certainly.

Evolving Solutions:
What factors play most heavily in developing a disaster recovery or business continuity plan, for example, government regulations, client contracts, or something more?

John “Traenk” Traenkenschuh:
Yes.

There is no single one factor that motivates even the busiest organizations.  It is a series of factors, acting randomly, that is creating ‘Perfect Storm’ conditions for compliance.  While you list excellent and compelling factors, we miss some of the more compelling.

Swine flu is a perfect example of an exposure that crops up out of nowhere.  Much like a 1970’s ship disaster movie, the premise seems almost ludicrous.  There is a new flu that can kill (and has).  It incubates in pigs, but leaps onto humanity like plague-filled fleas from centuries ago.  World Healthcare organizations predict a pandemic, in a vain attempt to predict the spread (in a small-world global community full of same-day flights).

When too little happens, organizations take this as a sign of needless panic and assume it’s ok to assume the best.  And then it comes, as assuredly as the ship’s belly-flop, everyone is backpedaling to try to fix the blame–instead of working to fix the problem.

I was a ‘Wave One’ responder after the Hurricane Andrew disaster.  Discussing Disaster Recovery conditions with those who have never helped in a sizeable Disaster Recovery event is so very frustrating.  Much like those BRP plans done without detailed knowledge of the underlying technologies, conversing with inexperienced BRP people is a lesson in futility.

It is rife with unfounded assumptions, those as ridiculous as “IP happens here”.  What happens when your organization calls in BRP workers from far away and your area has most road signs blown away?  When the switched landlines are gone, what mobile and cellular options have you lined up?  The water is yellow and brown; what do you do for your onsite workers, knowing local bottled water was sold out two days before?  You’re counting on local expertise to bridge the documentation gap; they have their own tragedies and family illnesses to work through.  What do you do now?

It is the raw unpredictability of today’s business events that compels us to work through the Disaster Recovery specifics, earlier, versus waiting for later.  Those organizations that aggressively prepare for BRP, with noted and experienced BRP experts, will survive.  Those that do not will find their instance of a Heartland data loss (or flu quarantine or…) may find survival impossible.

Evolving Solutions:
What tips would you offer for a business as it develops disaster recovery or business continuity plan for the first time?

John “Traenk” Traenkenschuh:
Surprisingly the most basic, the simplest advice, is seldom heard.  I would like to review a few basic ideas I share freely.  BRP need not be an overly expensive exercise in paperwork and meaningless reports.

a.     Talk with your business insurance professional – I am always surprised to find a business, no matter how small, that has no business insurance plan.

The Insurance Industry can only profit as your business avoids losses and/or lessens impact.  The right representative will take a personal interest in your business and keeping it free from grave exposures unconsciously assumed.  An onsite inspection, useful when calculating premium, can help find those crowded exit hallways that are littered with combustibles.  The inspections may be done for free in some cases; and if so, this can help you begin BRP efforts with low costs.

At some point, your organization will need to decide which exposures (and resultant costs) you will assume yourselves.  The biggest BRP issue too many face is:

•    Vagueness regarding what risks you still face,
•    Uncertainty concerning which of these you have self-insured, and
•    Indecision and inertia over which loss and impact mitigating steps your organization has signed up for.

b.    Create a BRP team within your organization – Let me guess…  You’ve already identified your ‘Goto Guy’ and have pinned all your hopes (and equal wrath) on him or her.  If this is true for your organization, this is ill advised for both political and for BRP reasons.  BRP is seldom popular when times are good and funding is available.  It becomes ‘hysterical over-reaction’ when times are tight and layoffs are ongoing.  Who in your organization is encouraging and funding BRP in these difficult times?

(Ironically, the downturns and layoffs are the most important times to have BRP plans updated!)

Your team needs to have members from a few influential parts of the organization, to ensure minimal funding at all times.  The same small team also ensures that the loss of any one member keeps BRP event handling continuous.  Seeing BRP as an IT-only exercise is another ‘biggest’ mistake an organization might make.  Until BRP is funded and supported by the organization itself, it simply will never be completed.

c.    Engage the services of a ‘Hired Gun’ – The stakes are high, very high.  The required skills involve IT, security, disaster recovery, virtualization, system and application design, IP networking, etc.  You will need an objective third party to bounce ideas off of.  At some point, the same party will need to coordinate disaster simulation exercises.  There is no reason to create forms and process flows for readiness—there is no time for that.  Implement, and not just plan, your BRP initiative!

Be ready for a series of tough questions that challenge your assumptions regarding how your organization operates.  Process improvements, including a painstaking inspection of externally sourced important applications and infrastructure, will be part of the package you hope to buy.  If there is no appetite for change, there can be none.  Meanwhile, how many of your best, most knowledgeable workers just retired or were laid off?  Change is inevitable despite our rejection or acceptance of it.

Hint:  BRP professionals who cannot work remotely or create suitable remote access systems, who insist on expensive and frequent onsite visits, these same people may be incapable of helping your organization’s flexibility during an actual BRP incident.

d.    Test (and Retest) your BRP technologies – Get the right support from the leadership and simply shut down a critically important system and/or application.  How quickly is control and coordination responsibility (and decision making authority) transferred to the correct group?  Were the promised backups performed reliably and were they available?  Hopefully, you will find that all went as planned.  If not, you will have the means to identify and remedy the problems.

I know that I have over-simplified what needs to go into a BRP plan and readiness effort.  Those organizations that are truly interested in improving their BRP readiness will act on steps this simple.  Others will simply laugh it off as hysteria, until it is too late.

Evolving Solutions:
Wild Card: Anything else you’d like to add?

John “Traenk” Traenkenschuh:
We started this interview with a brief glimpse at the business and IT improvements that are possible with virtualization technology, our second chance to exercise good governance and stewardship over the applications and information placed under our care.  The latter part of the interview discusses Business Resumption Planning and the many ways this field of study has grown, despite lack of interest from those organizations risking public and regulatory ridicule for simple mistakes.

Virtualization can provide focus for BRP efforts, all the while ‘Perfect Storm’ factors call on today’s BRP professionals to be ready for new and more challenging disaster scenarios.  In all of this, the issue is not a technical issue.  Instead, it is an issue with organizational efficiency and ability to face the challenges.  All the advice in this interview is provided freely, with no claims given to its sufficiency or perfection.  These views do not reflect the views of my employers, the organization publishing this interview, or any of my professional organizations with which I am associated.

I promise to do my best to respond to any and all queries left in response to this interview.  BRP, Virtualization, and Green IT are more than slogans to me.  They are a key to surviving in a radically changed business and social environment that seems more the stuff of nightmares than the evening news.

Business Continuity & Disaster Recovery Expert John “Traenk” Traenkenschuh

In a recent interview with Lawrence Webber, we discussed the ‘hows’ and ‘whys’ of business continuity and disaster recovery planning.  This week, our Data Center Leaders Interview Series drives home the importance of this topic during our two part interview with John “Traenk” Traenkenschuh.

Budding author, book editor, and Information Technology worker at three Fortune 100 companies, John “Traenk” Traenkenschuh’s insight into business continuity and disaster recovery planning comes courtesy of years of real world experience.

For his time spent introducing students to Microsoft’s Visual Basic, Traenk has been awarded the Microsoft Most Valuable Professional (MVP) designation since 2004.   He has also authored  VCP VMware 310 Cert Flash Cards as a late stage exam preparation tool.

In part one of our interview below, Traenk demonstrates the value of business continuity and disaster recovery planning through a look at both the evolution and future of  these solutions:

Evolving Solutions:
You have extensive experience in the fields of disaster recovery and business continuity.  How would you describe the evolution of recovery and continuity solutions since you first entered IT?

John “Traenk” Traenkenschuh:
Circular, the path from older Business Resumption Planning (BRP) options to today’s BRP options seems to have gone in a circle.

In times past, well defined applications were housed on well maintained and highly available hosts.  This design simplified identifying both the critical applications and the business data (and important external datastores) being acted on.  Flash forward a decade or more, and the emergent client-server model has us splattering app bits and pieces all over an increasingly ‘splashed to the four winds’ technical infrastructure.

The UNIX server acts as a client to the z/Series, fetching a copy of Accounts Receivable from some obscure PDS and then acting on it.  The new generation of Accounts Receivable data results are now posted to a Windows 2000 server, where someone’s copy of excel, running on a PC, performs data transformations that a staff assistant posts as authoritative graphs of the organization’s Accounts Receivable status.  And this mix of platforms and informal accesses is driving business decision making!

Indeed, everyone applauds the data without realizing the BRP issues:

•    How can we secure that data (and transformations that occur) across so many network and SAN paths?
•    What constitutes ‘safe storage’ in this ad hoc design?  Are any of the data generations ever reckoned back to the z/Series?
•    Which devices are now promoted to our high-priority computer/application list, those systems that MUST be restored by hour four of our BRP planning?  (And are we really comfortable with important data being manipulated and stored on the Staff Assistant’s laptop, possibly misplaced by absent-minded baggage handlers???)

Now flash forward, again, to 2009.  The right application of virtualization technologies can alleviate many of the harms we thought unsolvable just a few years ago.  We begin centralizing the technical infrastructures into a handful of virtualization hosts.  The mandate to virtualize means the company begins alerting and responding to the ad hoc IT flows that flooded our 90’s networks.

Throw in Desktop virtualization, and even those sore-point endpoints, the thousands of laptops and desktops winking on and off the internal network (so-called ‘Intraverse’), these are now backed up reliably.  (No one is saying goodbye to fat clients with the new scheme either.)

This is the core premise of virtualization technologies, that we might begin returning to required centralized technical and governance structures, structures that allow the organization to meet regulatory requirements, to cut costs, and to begin adopting a more green footprint as hundreds of dedicated computers are folded into a handful.

Evolving Solutions:
What disaster recovery and business continuity solutions do you see emerging in the next 5 years?

John “Traenk” Traenkenschuh:
If the term ‘solutions’ equals IT technologies, I think we start poorly?  BRP has always been a practices and procedures discussion; one implemented through technology to be sure, but one that has never been about technology, per se.

I believe, strongly, that the regulatory pressures and economic costs of today’s IT infrastructures require increased virtualization.  This will begin normalizing the infrastructure, the applications, the data (and access methods—maybe, more on that below), etc.  This will impact BRP in several fundamental ways:
•    Technical infrastructure BRP plans must no longer mirror a fractured infrastructure/Intraverse, one that includes all known and planned flavors of linux, a few Macs in the warehouse, Billy-bob’s mobile phone app, and who-knows-what ancient systems lingering in any one building’s computer center.  We lessen the options and force updates.
•    ‘Stealth’ processing and data results will be identified, making system- and application-prioritization more reliable.  Much like the show, “Cash in the Attic”, virtualization forces us to check into all the dataflows and systems , if we are to achieve our goal.
•    Flows that are difficult to manage may go outside the organization.  Increasingly, internal IT shops are no longer required to host every website nor to code up each and every workgroup-level Word macro.  Some of the processing, lurking in baling wire informal technologies that often run on volunteer hardware, these may need to go elsewhere for support.

I see governance to governmental regulations (and business partner practices) increasing the pressure to change.  If IT organizations cannot get a handle on internal pressures to [mis]manage application design and basic information access, away from longtime informal practices; at some point some-to-all IT services may be moved to Cloud Computing organizations, who will reduce an complex IT equation to a true Software as a Service (SaaS) offering.

At this time, IT organizations are in flux regarding whether to build an internal virtualization infrastructure or whether to vFarm IT Out to a third party.  There are compelling reasons for and against building or sourcing your vFarm.

There may be a middle ground:  ‘enhanced resources’, those you and I call ‘Consultants’.  These will be tasked with mapping legacy organizational practice with externally dictated Best Practices, with the idea that there can be a smooth transition plan to a world-class IT infrastructure.  However, that is an expensive course; and as a former consultant, I know that there are some very insular organizations that will not transition until they must.

In my mind, the Enron, and now banking crises, have made regulatory oversight of most organizations inevitable.  Payment Card Industry (PCI) compliance requires security testing with world-class tools.  Although imperfect, this system anticipates all organizations submitting to third-party security audits.

Please remember that I had seven great years in the Insurance industry, at a premier company.  Insurance works, providing premium sufficiency, because industry members manage to common processes that examine the risk exposures faced by all.  In fact, if I can add one small point, I’m shocked that the world of IT security metrics STILL does not have data-driven risk experience models as sophisticated as those used in the insurance industry.

Ask most insurance pro’s the relative claim value of the loss of mechanic’s finger, and they can arrive at a figure, no matter how obscure the facts.  Ask an IT person the relative value of a hairball analysis application, running at a pet food company with 23% market share, and you will wait, despite so much business data near at hand.

The Center for Internet Security (www.cisecurity.org) has an intriguing system of metrics announced recently.  Mitre.org’s CVSS is some help.  But overall, much remains mysterious, although commodity virtualization infrastructures and service offerings may bring us to a more common understanding of the worth of systems and their data, should a disaster occur.

If readers would like a short list of trackable technologies, those aiding BRP, let me offer this one:

•    Security Information and Event Management (SIEM) – Thanks to responsible vendors publishing long lists of security best practices, (Microsoft and others) and to the work of responsible security think-tanks covering security for multi-vendor environments (Center for Internet Security and SANS (@ www.SANS.org)); many organizations have enabled all the logging we can.  In debug mode at that!  This has made incident response as difficult as finding the proverbial needle in a haystack [of event information].  How can you prioritize BRP responses for the important applications, when you cannot separate the security wheat from the chaff?

•    Virtualization Security (known by many names and techniques) – Now that vFarms are hosting our applications, either on-site or off-site, we need to track what goes on at the virtualization ‘back-plane’.  The reoccurring fear is that a hacker can use a virtualized machine’s (VM) security weaknesses to attack the hypervisor, and then use it as gateway to other VM’s.  Another fear is the ‘rogue’ vFarm administrator who does all manner of bad things, accidently or maliciously.  In this world, the vendors to watch are those virtualization vendors with a long history of security prowess and competent tools AND those security vendors who offer solutions for the virtualization layer and for those VM’s needing their host security tracked and alerted (CA and eTrust and others).  The lack of security API’s in many virtualization packages is a leveling factor; few tools can operate at the backplane layer.  But to be sure, configuration audit and management is still possible.  As security API’s are provided by the vendors, being aligned to a solid security vendor will provide valuable.

•    Risk evaluation tools – As mentioned before, there is a fundamental fuzziness to security evaluation that makes risk mitigation difficult, if not dangerously off-the-mark.  Once regulations and cyber-security governmental appointments begin leveling the playing field, we’ll see new, improved risk models and companion tools that make risk evaluation less subject to personal and professional biases.  Maybe.

•    Green IT Movement – Complementary to BRP is the Green IT Movement.  Whether the computers gain electrical efficiency or we find ourselves growing a more extensive IT intraverse on fewer systems, these factors impact BRP directly.  Uninterruptable Power Supplies may be cut back, either because of fewer/more efficient computers OR because we do not want to proliferate an IT environment full of lead-acid batteries belching hydrogen fumes, possibly spilling sulphuric acid during a disaster.   Computer room temperature control units may be scaled back because of fewer computers, improving BRP focus.  I recently read a toilet paper wrapper that proudly proclaimed that the energy used during production was generated through windmills!  Expect all organizations to be encouraged to offer similar claims to environmental sensitivity—and for reasonable adjustments to be made to our BRP plans.

Part 2 of our interview with discussing Business Continuity & Disaster Recovery with John “Traenk” Traenkenschuh, discussing the factors guiding continuity and disaster recovery planning, and tips for getting a plan started, will publish later this week.

Storage Consolidation & Networking Expert Greg Schulz

As businesses in any industry grow, they often find themselves with both more data to mange and more costs to contain. Storage consolidation, networking and Green IT can all make for effective solutions if implemented in line with overall business needs and objectives.

Our Data Center Leaders Interview Series offers detailed advice on how companies can best keep synch needs and objectives with data center solutions, courtesy of Greg Schulz.  Schulz is founder of the independent IT analyst consultancy firm StorageIO Group and author of the books The Green and Virtual Data Center(1) and Resilient Storage Networks(2).

Below, we discuss with Greg how to begin development of a storage consolidation or networking plan, detail the benefits of Green IT and debunk the myth that business continuity and disaster recovery are only for the rich and famous:

Evolving Solutions:
In your book, Resilient Storage Networks, you speak of the growing need to store any piece of data and access it at any time.  At what point do businesses put themselves in danger of not being able to achieve this goal without a storage networking solution?

Greg Schulz:
The reality is that a networked storage, or, storage networking solution if you prefer, enables scaled connectivity either from a performance, accessibility, available or all of the above perspective beyond the limits of a dedicated direct attached connection.

Note that a direct attached connection could be a Serial Attached SCSI (SAS), Fibre Channel or iSCSI without a switch topology. Thus as more servers need to access shared storage than what can be provided by the native ports on a storage system, or, more storage is needed than what can be attached to the native ports on a server, a storage network becomes beneficial.

On the flip side to this, a storage network or networked storage solution can also exist in the form of a switch-less configuration, granted some of the storage networking police who prefer switches might not agree with the definition. For example, a large storage system with 8, 16, 32, 64, 128 or more Fibre Channel ports could be considered and are often marketed as a storage network solution.

The point is this, a benefit of a networked, or, storage network storage solution is the ability to scale performance, availability, capacity and connectivity beyond the limits of a smaller dedicated internal based storage solution.

Thus the emphasis should be on getting organizations to deploy shared storage that can grow with the business including attaching to larger switch based storage networks when and where needed.

Resilient Storage Networks by Greg Schulz

Evolving Solutions:
What do you recommend as a first step in the development of a storage consolidation or networking plan?

Greg Schulz:
Take a step back from the various technologies and tools. Have a clear understanding of business objectives and requirements to know how to make a linkage to the benefit of the solution and how the business is sustained or enhanced.

This helps to build a business case and then measure the ROI impact for future project enhancements particularly during tough economic times. Where this comes into play is being able to map to the business the technology that is needed to enable sustainability thus requires funding compared to those technologies or initiatives that would be nice to have or seen as discretionary and thus miss out on funding.

Also, keep a balanced approach to avoid tunnel vision on space capacity utilization. While there is a lot of rhetoric about the need to drive up storage or space utilization, don’t do it at the expense of degraded performance or introducing application instability or degraded quality of serve. Some systems have a low utilization to meet quality of service or other performance or service level requirements in which case the solution can be to move those applications onto faster performance storage in order to achieve affiance.

Keep in perspective what it is that you are looking to move or the problem being fixed. Thus, management insight tools are needed that can shed light not just on storage space capacity utilization, that also show performance and activity usage information. Find a balance between performance, availability, capacity and energy (PACE) o meet a given service level requirement and cost points.

These and others themes are discussed in more detail in my new book “The Green and Virtual Data Center” (CRC) available at Amazon and other venues.

Evolving Solutions:
How do green IT strategies synch with storage networking?

Greg Schulz:
Green IT strategies which are really about addressing and enabling business sustainability and growth by optimizing IT and data infrastructures have a strong synergy with efficiency enabling technologies including networked storage or storage networking as well as virtualization among others.

Most organizations do not buy and deploy networked storage, regardless of if SAN, NAS, Fibre Channel, FCoE or iSCSI just for the sake of buying it, that is unless they have a really good sales rep.  Instead, most organizations deploy storage networks as a means of improving efficiency and optimizing how IT resources are managed and used. Likewise, most organizations don’t or can’t afford to go green just for the sake of meeting PR or other initiatives and many organizations see green as being all about carbon footprints.

The reality is that most organizations have a need to address their power, cooling, floor-space or footprint as well as other environmental issues to support and sustain business growth, these are the other aspect of being green.

Ironically it is these issues that most organizations have been tasked with addressing and in many cases not making the connection that these are in fact green it issues that by addressing them, they in fact are being green. This paradigm is known as the green gap, thus tools, technologies and techniques including storage networks can help to improve on affiance and optimization enabling enhanced IT infrastructure resource management (IRM).

The Green and Virtual Data Center by Greg Schulz

Evolving Solutions:
In your experience, are small businesses or major enterprises more aware of the need for storage networking solution?

Greg Schulz:
There is a common myth that business continuity/disaster recovery (BC/DR) are issues and concerns only for the rich and famous meaning only for large enterprises or high profile financial organizations. The reality is that BC/DR along with general HA and data protection is affordable and the issues are applicable to organizations of all size.

Given the different size, scope and scale of these various sized organizations, there may or may not be the awareness of the benefits of specific technology including networked storage, thus there is an opportunity to show how data protection including BC/DR can be enabled and enhanced leveraging storage networks and other related technologies.

Likewise, there is an opportunity to leverage networked and shared storage for organizations of all sizes. At the high-end that can mean Fibre Channel and moving forward Fibre Channel over Ethernet (FCoE) or iSCSI in mid-sized environments along with NAS, while smaller environments can leverage low cost Fibre Channel, iSCSI, NAS or even shared, multi-host SAS based storage systems for small Exchange, SQL and VMware based clusters.

Thus tiered access for storage networks enables the most applicable technology to be mapped to the business needs of different environments. For example, a small environment can attach and share a dual controller RAID array such as the equipped with SAS/SATA disk drives attached via shared SAS, or larger environments share the same array across more servers with native iSCSI attachment, or even larger environments use the same array configured with Fibre Channel for either high performance scenarios.

Those are great example of tiered access to a tiered storage device enabling a networked or storage networked device for organizations of different sizes.

Evolving Solutions:
Leaving the company anonymous, what is the largest data disaster you’ve witnessed that could have been avoided with resilient storage networking?

Greg Schulz:
Not all disasters make the headline news, in fact, every day there are countless mini-disasters that due to configuration, best practices and how technology is used combine to contain these from expanding and rolling into major disasters.

In one situation it was a scenario where resilient components were interconnected thereby negating the benefits of no single point of failure (SPOF) resulting in downtime. In another scenario, lack of change control and configuration management, in another, it was a focus on capturing data in as small time slice interval as possible, only to not have captured all of the data that was still in memory thus resulting in an inconsistent recovery.

Another scenario was an outage where the IT systems were available; however no one could access the data as the network links were down including those from a secondary carrier who just happened to use the same common back-bone carrier.

Let’s not forget about scenarios where RAID and replication without point in time based snapshots or backups were confused with being a replacement for backup.

In those scenarios, data was in one case replicated to another site however when deleted at the primary, it was deleted at the secondary. In a related scenario, RAID was assumed to be the backup, that is until the entire RAID storage system became un-available including the snapshots that were stored on the device.

Don’t confuse or assume RAID and replication alone are replacements for backup and data protection as they need to be combined with some type of time recovery point based technique for complete or comprehensive data protection.

The common theme in all of these is that it’s not how much hardware or software, how many 9’s availability, it’s the people, processes and procedures including how they are configured to avoid, isolate or eliminate faults from spreading into disasters.

Thus, look for single points of failure, avoid being a penny wise and pound foolish. For example in the quest to save a few hundred or perhaps thousand dollars by skipping a second HBA or adapter on a server that is going to be used for virtualization and consolidation only to expose it as a single point of failure where multiple servers no exist (e.g. multiple eggs in one basket).

The bottom line is this, if something can happen it will, all technology will or can fail regardless of vendor marketing spin, however what differentiates various vendors is how they have learned from these faults or issues to configure around them leveraging best practices and their experience.

(1)The Green and Virtual Data Center is published by Auerbach-CRC.
(2) Resilient Storage Networks is published by Elsevier.

Learn more at www.thegreenandvirtualdatacenter.com or on twitter @storageio.

Business Continuity & Disaster Recovery Expert Lawrence J. Webber

Ensuring your business has a solid disaster recovery and business continuity plan in place isn’t just good practice, it can be a valuable sales tool.  With this in mind, we interviewed Lawrence J. Webber for the latest post in our Data Center Leaders interview series.

Along with Michael Wallace, Webber is one of the co-authors of the acclaimed The Disaster Recovery Handbook: A Step-by-Step Plan to Ensure Business Continuity and Protect Vital Operations, Facilities, and Assets.

Below, we discuss the reasons for your business to develop a disaster recovery and business continuity plan, how to get started, and how to use these plans as sales tool in front of prospects:

Evolving Solutions:
What factors play most heavily in developing a continuity plan, for example, government regulations, client contracts, or something more?

Lawrence J. Webber:
Disaster recovery plans are required for government regulations to protect stockholders from a company’s collapse in the face of a disaster (such as loss of a data center, etc.)  Their goal is to quickly restore essential company activities.  Non-essential activities are restored over time.

Business Continuity plans (actions in case of the failure of a significant component) are usually driven by customer requirements.  A reputation as a reliable supplier is valuable sales tool.

Companies providing Just-In-Time materials must have provisions to ensure that they can reliably deliver the expected goods even in the face of a problem.  This could be a need to set up a second assembly line, a second factory or to provide duplicate equipment for all process bottlenecks.

No matter how low your price – no one will buy if you cannot reliably deliver.

Evolving Solutions:
What are the most common misconceptions in regards to what a business continuity plan should or should not entail?

Lawrence J. Webber:
1.     Business continuity plans belong to the Business Continuity Manager.  Business continuity plans actually belong to the process owners, because if the process fails and the plan does not address the problem, it is that process manager who will under the management spotlight.

Since it is their plan, they must ensure it remains up to date and that team members know their roles.

2.    That the Business Continuity Manager (BCM) will “go write us a plan”.  This person coordinates the authoring of plans by others.  The BCM does not fully understand the processes of the Accounting Dept., the materials management group, the engineering team, etc.  Each group must fully participate in the process.  They often imagine the BCM will trot through their offices and magically write a workable plan for each.

Evolving Solutions:
What tips would you offer for a business as it develops continuity plan for the first time?

Lawrence J. Webber:
Don’t feel overwhelmed.  The plan only addresses restoring the critical business functions – perhaps 20% of the total.  Take it in stages.  ID what is most valuable, write a disaster recovery plan, and then write a business continuity plan.

It costs nothing to gather the basic information into one place:

•    Recall list for all personnel (phone numbers, emails, etc.).  Verify quarterly (preferably by calling them. Roster of all vendors, what they supply, and a 24 hour contact number.
•    List of support contracts (contacted via the vendor roster) along with what they support, hours of support, contract number, etc.
•    Build a calendar for when each contract expires
•    Keys to everything, including network cabinets, closets, passwords to servers, etc.
•    Ensure that ALL data residing on data center storage devices is backed up and then promptly moved off site to a secure storage area.  Verify that these back ups work, know who can recall the data and how to do it.
•    Identify critical IT systems, and the primary and secondary support person for each.
•    Ensure each person is on the recall list
•    Identify the critical components for each (servers, peripherals, etc.)
•    Ensure these items are covered by vendor support agreements

Evolving Solutions:
Susan Snedaker,  Principal Consultant with VirtualTeam and author of Business Continuity & Disaster Recovery Planning For IT Professionals, identified as the three biggest mistakes when developing a continuity plan as “Not Creating A Plan,” “Not Getting Executive Buy-In,” and “Not Getting The Right People In The Room.”  What would you add to this list?

Lawrence J. Webber:
False confidence that once a plan is written, you are safe.  It must be regularly tested (perhaps quarterly) so that everyone knows their roles and that the plan reflects the current processes.  A plan sitting on a shelf is a snapshot in time.

Processes change, so do process tasks and staffing.  The document does not magically change by itself, and often no one bothers to inform the Business Continuity Manager.

Evolving Solutions:
How would you measure the chance of a newly launched company’s success, with or without, a disaster recovery or business continuity plan?

Lawrence J. Webber:
A disaster plan and a business continuity plan are only called into action when something goes wrong.  A new company with potentially excess capacity can disguise a disruption from a customer.  However, a well run company, tightly staffed, cannot disguise a disruption.  At best, they are tempting fate.

Evolving Solutions:
Wild Card: Anything else you’d like to add?

Lawrence J. Webber:
Disaster recovery is all cost.   Like insurance, you pay and pay but usually never need it (ie no disasters strike).

Business Continuity planning provides payback in resilient processes which result in more reliable cost estimates and product/service delivery. Green and Lean initiatives (such as virtualizing servers) also shortens recovery time.

Green IT Expert John Lamb, PhD

Green IT is a subject we have dedicated significant (virtual) ink to on this blog.  In discussing Green IT, we strive to illustrate the link between eco-friendly energy management and significantly lower business costs.

Helping us illustrate this link is John Lamb, PhD, as he contributes his insight to our Data Center Leaders interview series.

Lamb is a Senior Certified IT Architect with IBM Global Services.  He has authored over 50 technical whitepapers and four books including, “The Greening of IT:  How Companies Can Make A Difference For The Environment.”(1)

Below, we discuss the steps needed to make your data center eco-friendly, the cost savings that can be expected, and the future of green IT:

Evolving Solutions:
What tips would you offer for business seeking to reduce data center costs through green initiatives?

John Lamb:
A very straightforward process, and probably the most significant improvement data center management can make, is to use the standard server refresh policy (which is typically every four years) to move to virtual servers.  Virtual data storage would follow.

Of course, the very first step would be to “get the facts” and diagnose where your data center energy is being used. In addition to the diagnose step, four other steps are to measure/manage, cool, virtualize, and build.

Additional steps such as communications/appointing an energy czar, analysis of application efficiency, and making use of rebates and incentives could further help improve the business case for going green.

Improving energy management is an ongoing endeavor. Improving energy efficiency requires focusing on a number of areas: the IT equipment, the data center facility, and the on-going energy management. The five-step process is a way to show a set of actions across all these areas. The idea should be to have continuous improvement.

Evolving Solutions:
Are there inherent dangers in trying to make your data center too cost efficient through green IT?

John Lamb:
There could be the “gold plating” syndrome. An engineer or IT architect can actually try to go too far in reducing energy use. The basic business case with a focus on a good return on investment (ROI) always needs to apply.

Back in the late seventies we designed solar heating for several IBM buildings and actually implemented a few solar heating projects. We could realize a significant reduction in energy use, but if the payback period is 20 years and the life of the solar heating system is only 20 years, then that’s not a good investment.

However, for data centers the cost saving incentives are so great companies have significant motivation from a financial standpoint to go green.

The Greening of IT by John Lamb, PhD

Evolving Solutions:
Your book, The Greening Of IT, describes how IT vendors are touting eco-friendly policies such as carbon-neutral computing in their sales pitches.  With corporations typically driven more by bottom-line factors, do you fear taking the “green” angle may cause sales pitches to fall on deaf ears?

John Lamb:
I believe most companies do feel a corporate responsibility to help the environment.  However, the best motivator to get started – whether it’s a company or an individual – is to show the economic benefits of reducing energy use.

Let’s face it, if a company or individual can be shown methods to cut energy use and save money by following best practices, that’s always a great motivator. If a company can be shown that along with cost savings the company is also helping the environment, then we have a real “win-win” scenario.

So, to answer your question, the primary goal should be to cut costs through energy efficiency.  That goal will automatically lead to the goal of helping the environment.

Evolving Solutions:
Toby Velte, Global Technology Strategies with Microsoft, describes how he helps to ensure Green IT initiatives are funded by always relating projects to the pressures of capitalism, rather than the pressures of altruism.  Do you find this to be true across the board, or have you seen some firms consider start to implement green IT purely from a sense of corporate responsibility?

John Lamb:
I agree with Toby. The first and best motivator to go green is to show the financial benefits from the energy savings.

As mentioned in the response to the previous question, after showing the economic benefits it’s a great idea to also show the benefits to the environment.  Then we have a win-win situation for both the CFO and the executives who want to show corporate responsibility with improvements to the environment.

Evolving Solutions:
How would you recommend selling the idea of green IT to upper management who see it as little more than a fad?

John Lamb:
I’d recommend giving upper management some real life case study examples of the money that can be saved.

A typical US data center of 25,000 square feet will use approximately $2.6 million in energy costs per year at 12 cents per KWH. Improvements in energy management can save up to 50% of those costs.  Over a million dollars in savings is typically a motivator that will drive sufficient interest.

If upper management can be given references along with business case details of other companies that have experienced significant energy cost savings by going green that should do the trick.

All companies will become serious about reducing energy through green IT once they realize the significant cost savings possible even by initially only going after the low hanging fruit.

Evolving Solutions:
Anything else you’d like to add on Green IT or data center cost avoidance?

John Lamb:
Two emerging technology areas for green IT that intrigue me are the use of fuel cells to power data centers and the use of private cloud computing for the ultimate in server and data storage virtualization.

Fuel cells are not new – they powered the space capsules that carried men to the moon. Hydrogen powered fuel cells are very environmentally desirable since the only output, in addition to energy, is water.

The problem is in obtaining the hydrogen.  Currently hydrogen is usually produced through a very energy intensive process using natural gas and immense amounts of electricity.  When technological breakthroughs allow us to produce hydrogen efficiently, then fuel cells for data center energy will be a significant step forward.

Cloud computing allows companies to move to virtualization of all computing systems and to very high levels of utilization. Cloud computing – both public and private – is evolving quickly and is already having an impact on green IT.

(1) Publisher disclaimer: “The Greening of IT: How Companies Can Make a Difference for the Environment” by John Lamb; ISBN 0137150830, published April 2009 by IBM Press (Copyright 2009 by International Business Machines Corporation). To view a sample chapter, please click on “Sample Pages”: www.ibmpressbooks.com/title/0137150830”

VMware & Virtualization Expert Edward L. Haletky

When asked for technologies key towards sustained IT cost reductions, data center leaders commonly cite server virtualization technologies.

Our Data Center Leaders interview series digs deeper into this subject today as we discuss VMware virtualization with AstroArch founder Edward L. Haletky.

Haletky is a leading expert on VMware and virtualization who has authored the books VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers and the upcoming Virtual Infrastructure Security: Securing ESX and the Virtual Environment.

Haletky shares insight below on virtualization best practices, including how to identify and eliminate security threats common to virtual platforms:

Evolving Solutions:
Your book, VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers, offers VMware deployment tips and best practices.  What top three tips would you offer to businesses who’ve just implemented server virtualization through VMware?

Edward L. Haletky:
Now that the implementation is complete, it is time to review everything to make sure no changes are necessary. In some cases, these changes could mean a reinstall due to a change in usage or the plan.

You can now gather performance data to see how things are working and adjust the system appropriately.  This is the time to make sure your investment in virtualization succeeds. Verify memory limits, perhaps you have allocated more memory than what is actually used, verify Disk and Network IO, you may need to adjust this as well by adding new LUNs or more pNIC to the vSwitch, etc.

Form a Team that comprises the Security, Storage, Server, Network, and Virtualization Administrators to discuss issues as they come up. Sort of an advisory board of sorts within the company. With virtualization, the traditional siloed approaches do not work very well.

When considering new hardware, always choose something that is on the Hardware Compatibility Lists and nothing off them.

Evolving Solutions:
When asked for technologies designed to reduce costs in larger data centers, Cisco’s Omar Sultan cited virtualization technologies such as VMware and Hyper-V. How soon after implementation of these technologies can businesses expect to see cost savings?

Edward L. Haletky:
That depends on quite a few things, but most companies see an immediate lower consumption in power and possibly even cooling.  Those are always the big savings and immediate ones.  Cost savings also occur when hardware is to be updated as you are updating less hardware but license costs tend to eat into those savings.

Over time more items will be virtualized and a new baseline for cost savings will be created that already includes virtualization.

The real savings will end up being in efficiency.

Evolving Solutions:
Your upcoming book, Virtual Infrastructure Security: Securing ESX and the Virtual Environment promises to help identify and mitigate security related threats in all VMware platforms.  What would you identify as the single largest security threat present in VMware platforms today?

Edward L. Haletky:
This is a tough one, but it can boil down to the fact that currently virtualization security does not encompass the entire virtual environment but concentrates just on virtualization host security.

There is quite a bit more to virtualization than just a hypervisor to consider: there is management, backup, storage, clustering, and virtual networking.

In addition, security is often considered a bolt-on or after thought when it should be considered from the very beginning, when you are architecting and designing your virtual environment.

Evolving Solutions:
In regards to leaving themselves open to security threats, what are the biggest mistakes companies make after implementing a virtualization initiative and how can they be avoided?

Edward L. Haletky:
Many companies bolt on security instead of design/architect it in from the beginning.

That aside, the biggest error I see is the use of a flat network for management, IP storage, and VMotion. These three networks should actually be separate from each other and the normal production networks using firewalls and perhaps separate physical switches.

The other item that comes to mind on virtual networking is the level of trust in VLANs. This is not a security construct but people use it as such.

The other issue that comes up is to overlook aspects of storage security such as how backups are made.

In general, most people feel that they cannot be attacked and that they are safe from attack due to having an external firewall. Until a Penetration Tester comes in and shows how false that is, ignorance is bliss.

Evolving Solutions:
VMware vSphere 4 is bringing virtualization to small businesses. How have small businesses reacted to this opportunity promising to improve data center efficiency?

Edward L. Haletky:
vSphere 4 is not really doing that, it was done when VI 3.x was released as well as when VMware Server and ESXi were offered for free. Yes vSphere 4 builds on this, but the promise was made when ESX v3 was released.

Over the last few years I have seen more and more small organizations like Doctor’s offices turning to virtualization. They do this to cut their electrical costs. The last place a Doctor tends to invest is into IT.

I think vSphere 4 as critical new tools for the Enterprise and some specific SMBs, but in general, they like what was already available. Now that is improved.

Evolving Solutions:.
Wild Card: Anything else you’d like to add?

Edward L. Haletky:
Visit www.astroarch.com/wiki/index.php/VMware_Virtual_Infrastructure_Security for the latest information on my latest book “VMware vSphere(TM) and Virtual Infrastructure Security”, which is now available in a pre-edited version on Rough-Cuts.