Big Data in 2016: Hadoop

Posted on

Ken Hess of ZDNet reports on the results of a recent big data trends survey that focused on Hadoop use. The survey asked 250 data architects, IT managers, developers, BI and data analysts and data scientists on what they expect in 2016.

The overall theme of the survey results writes Mr. Hess is, “the move away from Hadoop experimentation into full production with big data analytics.”  The big “three” trends identified in his article were:

  • Apache Spark production deployments
  • Conversion from other platforms to Hadoop
  • Leveraging Hadoop for advanced use cases

70% of respondents were interested in Apache Spark. Mr. Hess explains, “The two primary factors in this interest in Spark is that it is easy to deploy and its speed.” The conversion from older platforms to Hadoop is also interesting. When asked what they expect to gain from switching to Hadoop the respondents cited the following:

  • 63% expect Hadoop to increase business and IT agility
  • 55% expect to improve operations and reduce costs
  • 51% expect it to allow them to give business users more access

Also, Mr. Hess reports, “More than half the respondents view Hadoop as a way to innovate by using social media data and data from IoT sources.”  One of Hadoop’s strengths is its ability to bring data together. Companies used to struggle with just combining internal data that was often in siloes. In today’s world, companies must also find ways to combine and process external data with their internal information and that is where Hadoop  can come in to help.

As January winds down, what does your team hope to accomplish on the big data front in 2016 and what tools are you most interested in learning more about?

Big Data Analytics Trends for 2016

Posted on

Today let’s look at what is in the news for big data analytics trends and what industry leaders are predicting for 2016.

First, David Weldon of Information Management writes, “Data analytics remained one of the most important areas of technology investment throughout 2015, and industry analysts agree the same will hold true in 2016.”

Big data trends from his article are:

  • More businesses will seek out value in all data
  • Platforms that support “data in motion” will evolve
  • Demand for “big data made easy” will drive platforms to simplify
  • Hadoop will be used for more enterprise critical workloads

Venture Beat reports the growth in machine learning will be a key driver in the big data industry, “Machine analytics will be the fastest growing area of big data, which will have CAGR greater than 1000%. As digital transformation grows, so does the reliance on new software and architectures. Today, software is not only driving business processes, but entire business models, and the need to manage, monitor, and troubleshoot applications in real-time has never been more critical. Thus, the need for speed, full-stack visibility, and agility — all in real-time — is the true business demand underpinning the growth of machine data analytics.” Machine learning will drive changes in DevOps, bring to light the importance of log management and new software-based architectures will grow that better utilize the processing power of public and private clouds. Finally, business intelligence will change from “rear-window” analytics to continuous big data analytics in real-time.

Hema Reddy of IBM’s Big Data Hub writes, “Big data is the world’s newest natural resource. The value of big data is self-evident, and the need for powerful analytics capable of delivering that value is pressing in the business environment. Business leaders no longer question whether value is to be found in data; instead, they wish only to learn how to extract that value to understand their customers and meet critical business needs.”  In 2016 companies will continue to look for platforms that allow them to extract value from analytics including rich media such as video and images. Predictive analytics will continue to be important as a means to drive efficiency and finally big data growth in the cloud will continue.

Data is everywhere and in 2016 more companies will find better ways to capture it and analyze it to drive innovation.

The Enterprise Mainframe and Big Data

Posted on

So what is next for the mainframe? Ok, maybe you haven’t asked yourself that in awhile or perhaps you didn’t think it was necessary, but according to Ken Hess of ZDNet big data may be in its future.

Mr. Hess examines a survey of IT pros. He writes, “Based on the survey results, the mainframe may be the key to closing the big data processing gap and supporting the latest data delivery technologies. Over 89 percent of respondents indicated that mainframe’s key benefit in the next year is its Big Data processing horsepower through CICS, DB2 and WebSphere, and more than a third see it as an opportunity to provide data to big data platforms such as IDAA, Netezza, Splunk, Spark, Oracle, Teradata and Hadoop.”  Key findings from the survey include:

  • 69% of respondents ranked the use of the mainframe for large-scale transactions as “very important”
  • 79% still analyze real-time transactional data from the mainframe with a tool that is also on the mainframe, but survey respondents are also using outside tools such as Hadoop (8.6%)

The survey also asked about the strengths of using a mainframe for analysis:

  • 67% pointed out that a key strength of the mainframe is its ability to integrate with other platforms
  • 82% felt the mainframe provides good security
  • 83% of respondents also cited availability as a key strength

Mr. Hess further points out that companies like IBM are also launching systems that help mainframes connect and utilize outside big data analysis tools like Hadoop.

So when you think big data, do you also think about your mainframe?  This article further underlines the importance of a thorough evaluation of your company’s unique resources and systems against outside options in order to develope a big data solution that will meet your needs.

Big Data Put into Practice: Patient Care & Costs

Posted on

Greg Freiherr writes for CIO, “Mining massive collections of patient data has healthcare poised to improve patient outcomes while holding down costs.”  Today let’s look at two real-life uses of big data in the healthcare industry that improve patient care and costs.

The work of data scientists at Penn Medicine in Philadelphia was highlighted by Ken Terry on CIO.  Penn Medicine has data in the amounts of petabytes and it is using big data tools to analyze and create better care pathways. The team not only sets out new pathway ideas, but also tests with patients and “feeds” the results back into their algorithms to learn and recalculate. Mr. Terry reports that so far the team has improved the hospital’s ability to predict what patients are at a high risk of developing sepsis.  The team can identify patients a full 24 hours sooner than it could prior to using big data methods.

Jessica Davis reports for Information Week that Blue Cross Blue Shield (BCBS) is using big data to tackle healthcare quality and costs, “the Axis big data initiative at Blue Cross Blue Shield Association incorporates more than $350 billion in annual claims, 36 million provider records, and more than 700,000 BCBS patient reviews, all with the goal of helping healthcare consumers make informed decisions.” Axis will be a reference tool that allows consumers to compare costs by geography but also goes one step further and looks at the costs of pre-procedure and follow ups – providing a “total cost price reference.”  All 36 BCBS members are now contributing data making it the only healthcare system to have data from every ZIP code in the U.S.

The Penn Medicine example focused on patient care and Blue Cross Blue Shield focused on evaluating costs are two examples of the power of big data to benefit the healthcare industry in numerous ways.

Big Data Analytics and Variety

Posted on

Chris Gillar of IBM Big Data writes, “when you choose analytics solutions, you’ll likely have many diverse needs, all of which either need to be addressed now or will someday need to be addressed. Moreover, because even a single platform can offer considerable variety, getting the variety you need all at once has never been simpler. But you must also take care that the variety offered is the right variety.”

In a recent article he discusses the importance of choice in your big data analytics solution.  He identified five common needs and points out businesses will accommodate these needs in different ways which calls for a platform that offers variety and choice. Here are the common capability needs:

  • Support of a Hadoop-based solution.  With open source frameworks becoming more popular there are also certain capabilities that your company may or may not have that need to be there. Mr. Gillar highlights the need for strong SQL on Hadoop and full integration and support for R, text analytics and more
  • Support business intelligence and performance management whether on-premise, cloud or hybrid
  • Support predictive and prescriptive analytics. Mr. Gillar writes, “But you must look for capabilities such as the ability to perform analytics regardless of whether data is at rest or in motion, as well as the ability to automate data preparation. These two features in particular simplify analytics and make it inclusive”
  • Support of real-time analysis and solution development. Data is moving fast and your company needs to be able to analyze and react to it.  Your big data solution should offer a variety of capabilities so you can choose what makes most sense for your company
  • Support business users. Your solution must not only be powerful behind-the-scenes but it must also be usable by non technical employees across the organization

Seeking out a big analytics solution that brings you variety and choice will better allow you to meet your company’s needs now and in the future.