Data deluge « A Software Insider's Point of View

Posts Tagged ‘Data deluge’

Tuesday’s Tip: Focus On The Business Outcomes, Not Technology With Big Data

Published on February 26, 2013 by R "Ray" Wang

The Why Behind Big Data Starts By Asking What’s The Business Outcome

So organizations have lots of data. New techniques have emerged to correlate big data. Enamored by the potential of big data, leaders are now reinvesting in technologies to find hidden nuggets of insights with the business goals of:

Mitigating regulatory risks
Identifying operational efficiencies
Improving revenue growth
Creating market differentiation
Expanding the brand presence

These big data use cases often follow the business hierarchy of needs, which are based on concepts pioneered by Maslow (see Figure 1). More importantly, a key question in big data has been to ask the right question.

Figure 1. The Business Hierarchy of Needs Drives Many Big Data Use Cases

An Information Flow Approach Moves The Discussion From Data To Decisions

Unfortunately, the problem is most organizations start by talking about outcomes and then get mired in the technologies to achieve these outcomes. Big data technologies include advanced business analytics, application of existing technologies such as data warehousing and business intelligence. In many cases, application of decision automation, semantic technology and collaborative tools are also needed. Yet, from Data to Decisions requires the integration of quite a few disciplines.

Data to decisions is about taking data sources, transforming them into useful information, gathering key insights, and then making the right decisions (see Figure 2). Data sources, information, and orchestration belong in the realm of IT and hopefully will be delivered via the cloud. Insight, decisions, and actions are line of business driven areas which deliver the most value add:

Data sources. Expect a mix of structured, semi-structured, and lots of unstructured.
Information and orchestration. The mix of information types include physical, virtual, machine, and contextual.
Insight. Information translated to insight considers performance, deduction, inference, and prediction.
Decisions and actions. The outcomes are driven from next best action, prevention, suggestion, and even no action.

Figure 2. The Flow From Data To Decisions

More…

Tuesday’s Tip: The Big Question In Big Data Is…What’s The Question?

Published on June 19, 2012 by R "Ray" Wang

All The Current Talk Of Big Data Technology Misses The Point

The hype around big data has crescendoed to the levels of SOA in the early 2000′s, cloud in the late 2000′s, and social in the past few years. Unfortunately the hype is creating three main pitfalls:

A morass of confused definitions. In fact a quick survey of any educated audience, yields a multitude of definitions. Some folks see big data as large data sets and data warehouses, others see big data as code for analytics and BI. Many see the output of big data as infographics or the hardware behind the support of big data. The V’s of big data continue to expand from volume, velocity, and variety to include veracity, viscosity, and virality. Some folks even have 16 V’s in their definitions.
Solutions confusion among buyers. A technology vendor land grab for mind share with big data is happening now the same way everyone adopted cloud. Hardware vendors now enable big data. Storage providers now deliver big data solutions. Integration vendors provide plumbing and intelligent connections for big data. Analytical vendors now all support big data. Some folks like to confuse Hadoop with big data. Everyone has a solution, just not the solution a buyer thinks they need. Confused capabilities continue to proliferate amidst a lack of good customer references. Customers feel the chaos.
Discussion on technology options not business problems. The discussion about big data has evolved into a technology conversation not a business value or transformation conversation. Clients immediately talk about products and technologies without defining the problem to be solved. Technology investments take over the discussions on solution development.

Recommendations: Focus On the Questions To Ask, Not The Answers.

More…

Monday’s Musings: Why Are Innovative CIO’s Betting Less On Cloud And Virtualization?

Published on May 7, 2012 by R "Ray" Wang

Innovative CIO’s Betting On Disruptive Technologies That Impact Enterprise Business Value

In the Four Personas of the Next Gen CIO published March 3, 2012, four personas of the CIO were identified: Chief Infrastructure Officer, Chief Integration Officer, Chief Intelligence Officer, and Chief Innovation Officer (see Figure 1). This research of 79 progressive CIO’s identified the key projects for each of the personas. As part of the survey, respondents were asked what key disruptive technologies would make an impact in the enterprise in the next year.

Figure 1. The Four Personas Of The Next Generation CIO

Source: Constellation Research, Inc.

In Constellation’s latest update (to be published May 2012), 105 innovative CIOs participated in the survey. The results indicate a shift away from cloud (56.4%-2012) and virtualization (29.6% – 2012) to mobile (60.2%-2012) and big data and analytics (48.7%-2012) (see Figure 2). Despite being the top projects in 2011, the drop in priority of virtualization (51.9%-2011) and cloud (69.6%-2011) doesn’t reflect the lack of interest. In fact, these projects have matured and innovative CIOs have now prioritized the next wave of innovation.

More…

Monday’s Musings: Beyond The Three V’s of Big Data – Viscosity and Virality

Published on February 27, 2012 by R "Ray" Wang

Revisiting the Three V’s of Big Data

It’s time to revisit that original post from July 4th, 2011 post on the the Three V’s of big data. Here’s the recap:

Traditionally, big data describes data that’s too large for existing systems to process. Over the past three years, experts and gurus in the space have added additional characteristics to define big data. As big data enters the mainstream language, it’s time to revisit the definition (see Figure 1.)

Volume. This original characteristic describes the relative size of data to the processing capability. Today a large number may be 10 terabytes. In 12 months 50 terabytes may constitute big data if we follow Moore’s Law. Overcoming the volume issue requires technologies that store vast amounts of data in a scalable fashion and provide distributed approaches to querying or finding that data. Two options exist today: Apache Hadoop based solutions and massively parallel processing databases such as CalPont, EMC GreenPlum, EXASOL, HP Vertica, IBM Netezza, Kognitio, ParAccel, and Teradata Kickfire

Velocity. Velocity describes the frequency at which data is generated, captured, and shared. The growth in sensor data from devices, and web based click stream analysis now create requirements for greater real-time use cases. The velocity of large data streams power the ability to parse text, detect sentiment, and identify new patterns. Real-time offers in a world of engagement, require fast matching and immediate feedback loops so promotions align with geo location data, customer purchase history, and current sentiment. Key technologies that address velocity include streaming processing and complex event processing. NoSQL databases are used when relational approaches no longer make sense. In addition, the use of in-memory data bases (IMDB), columnar databases, and key value stores help improve retrieval of pre-calculated data.

Variety. A proliferation of data types from social, machine to machine, and mobile sources add new data types to traditional transactional data. Data no longer fits into neat, easy to consume structures. New types include content, geo-spatial, hardware data points, location based, log data, machine data, metrics, mobile, physical data points, process, RFID’s, search, sentiment, streaming data, social, text, and web. The addition of unstructured data such as speech, text, and language increasingly complicate the ability to categorize data. Some technologies that deal with unstructured data include data mining, text analytics, and noisy text analytics.

Figure 1. The Three V’s of Big Data

Contextual Scenarios Require Two More V’s

In an age where we shift from transactions to engagement and then to experience, the forces of social, mobile, cloud, and unified communications add two more big data characteristics that should be considered when seeking insights. These characteristics highlight the importance and complexity required to solve context in big data. More…

Monday’s Musings: The Three V’s of Big Data

Published on July 4, 2011 by R "Ray" Wang

The Three V’s Traditionally Define Big Data

Traditionally, big data describes data that’s too large for existing systems to process. Over the past three years, experts and gurus in the space have added additional characteristics to define big data. As big data enters the mainstream language, it’s time to revisit the definition.

Volume. This original characteristic describes the relative size of data to the processing capability. Today a large number may be 10 terabytes. In 12 months 50 terabytes may constitute big data if we follow Moore’s Law. Overcoming the volume issue requires technologies that store vast amounts of data in a scalable fashion and provide distributed approaches to querying or finding that data. Two options exist today: Apache Hadoop based solutions and massively parallel processing databases such as CalPont, EXASOL, GreenPlum, HP Vertica, IBM Netezza, Kognitio, ParAccel, and Teradata Kickfire
Velocity. This characteristic describes the frequency at which data is generated, captured, and shared. The growth in sensor data from devices, and web based click stream analysis now create requirements for greater real-time use cases. The velocity of large data streams power the ability to parse text, detect sentiment, and identify new patterns. Real-time offers in a world of engagement, require fast matching and immediate feedback loops so promotions align with geo location data, customer purchase history, and current sentiment. Key technologies that address velocity include streaming processing and complex event processing. NoSQL databases are used when relational approaches no longer make sense. In addition, the use of in-memory data bases (IMDB), columnar databases, and key value stores help improve retrieval of pre-calculated data.
Variety. A proliferation of data types from social, machine to machine, and mobile sources add new data types to traditional transactional data. Data no longer fits into neat, easy to consume structures. New types include content, geo-spatial, hardware data points, location based, log data, machine data, metrics, mobile, physical data points, process, RFID’s, search, sentiment, streaming data, social, text, and web. The addition of unstructured data such as speech, text, and language increasingly complicate the ability to categorize data. Some technologies that deal with unstructured data include data mining, text analytics, and noisy text analytics.

The Bottom Line: Start With Your Business Objectives

In Stephen Covey’s book, Seven Habits of Highly Effective People, he starts with a saying, “Begin with the End in Mind”. For big data projects, ask the key questions. What patterns will you uncover that will change how you go to market or address fraud? Can you apply sentiment and location to create new customer experiences. What additional insights can help you create new and disruptive busienss models? Big data is just a technology and tool. How you apply this tool to your business models and objectives will determine whether big data is a luxury or a necessity.

Your POV

What business problem will require you to start with Big Data? What are the key outcomes? Where do you expect to move the needle? Add your comments to the blog or send us a comment at R (at) SoftwareInsider (dot) org or R (at) ConstellationRG (dot) com

Resources

Reprints

Reprints can be purchased through Constellation Research, Inc. To request official reprints in PDF format, please contact Sales .

Disclosure

Although we work closely with many mega software vendors, we want you to trust us. For the full disclosure policy, stay tuned for the full client list on the Constellation Research website.

* Not responsible for any factual errors or omissions. However, happy to correct any errors upon email receipt.

Copyright © 2001 -2011 R Wang and Insider Associates, LLC All rights reserved.
Contact the Sales team to purchase this report on a a la carte basis or join the Constellation Customer Experience!

Monday’s Musings: Using MDM To Build A Complete Customer View In A Social Era

Published on May 9, 2011 by R "Ray" Wang

Customers Have Evolved… Has Your Organization?

Right now customers and prospects probably ignore your organization’s marketing messages because mass marketing campaigns lack relevancy. Right now most customers answer each other’s questions because your customer service and support agents lack the authority or knowledge to resolve issues. Right now prospects ask each other what they think about a company’s product or service because most organization’s sales professionals lack credibility.

Consequently, organizations face immense challenges in influencing prospects and customers as three forces drive the changing dynamics in customer engagement (see Figure 1).

Trust not financial performance is the new social currency. Trust drives influence, engagement, and relationships. People and organizations must earn trust through their actions across their relationships. Trust can be expended to gain influence, create engagement, and foster relationships. Trust can be taken away through lack of credibility, bad behavior, and dishonesty.
Increase in social media adoption moves beyond fad. Social media adoption is a cultural shift not a fad. The growing preference for engagement through social channels drives new relationship models. Social has moved beyond the tipping point. How social evolves and permeate our lives is the question.
Failure of CRM efforts to engage and influence. Traditional CRM focused on management versus engagement. CRM initiatives barely addressed customers and mostly ignored relationships. Projects focused on manager convenience instead of employee empowerment. More importantly, systems supported transactions not relationships.

Figure 1. Three Forces Drive The Changing Dynamics In Customer Engagement

Organizations Must Address Their Data Challenges To Gain A Strategic Advantage

More…

Research Report: Constellation’s Research Outlook For 2011

Published on January 4, 2011 by R "Ray" Wang

Organizations Seek Measurable Results In Disruptive Tech, Next Gen Business, And Legacy Optimization Projects For 2011

Credits: Hugh MacLeod

Enterprise leaders seek pragmatic, creative, and disruptive solutions that achieve both profitability and market differentiation. Cutting through the hype and buzz of the latest consumer tech innovations and disruptive technologies, Constellation Research expects business value to reemerge as the common operating principle that resonates among leading marketing, technology, operations, human resource, and finance executives. As a result, Constellation expects organizations to face three main challenges: (see Figure 1.):

Navigating disruptive technologies. Innovative leaders must quickly assess which disruptive technologies show promise for their organizations. The link back to business strategy will drive what to adopt, when to adopt, why to adopt, and how to adopt. Expect leading organizations to reinvest in research budgets and internal processes that inform, disseminate, and prepare their organizations for an increasing pace in technology adoption.
Designing next generation business models. Disruptive technologies on their own will not provide the market leading advantages required for success. Leaders must identify where these technologies can create differentiation through new business models, grow new profit pools via new experiences, and deliver market efficiencies that save money and time. Organizations will also have to learn how to fail fast, and move on to the next set of emerging ideas.
Funding innovation through legacy optimization. Leaders can expect budgets to remain from flat to incremental growth in 2011. As a result, much of the disruptive technology and next generation business models must be funded through optimizing existing investments. Leaders not only must reduce the cost of existing investments, but also, leverage existing infrastructure to achieve the greatest amount of business value.

More…

Monday’s Musings: Why Next Gen Apps Must Improve Existing Activity Streams

Published on August 30, 2010 by R "Ray" Wang

Upcoming Data Deluge Threatens The Effectiveness Of Activity Streams

Activity streams, best popularized by consumer apps such as Facebook and Twitter, have emerged as the Web 2.0 visualization paradigm that addresses the massive flows of information users face (see Figure 1). As a key element of the dynamic user experiences discussed in the 10 elements of social enterprise apps, activity streams epitomize how apps can deliver contextual and relevant information. Unfortunately, what was seen as an elegant solution that brought people, data, applications, and information flow into a centralized real-time interface, now faces assault from the exponential growth in data and information sources. In fact, most people can barely keep up with the information overload, let alone face the four forces of data deluge that will likely paralyze both collaboration and decision making (see Figure 2):

Massive activity stream aggregation by enterprise apps. Every enterprise app seeking sexy social-ness plans one or more social networking feeds into their next release. The mixing and mashing of personal and work related feeds will leave users confused about context and lower existing signal to noise ratios. Yet, proliferation will continue as users seek to bring aggregated sources of information into one centralized feed.
Explosive growth in the Internet of Things (IOT). Beyond just device to device communications, the web of objects, appliances, and living creatures through wired and wireless sensors, chips, and tags will drive most of the growth in the internet in the next 5 to 10 years. With an estimated 100 billion net-enabled devices by 2020, these networks seek to discover activity patterns, predict outcomes, and monitor operational health. The massive amounts of sensing data driven into systems will not only overwhelm users, but also handicap the performance of today’s data warehouses, analytics platforms, and applications.
Flood of user generated content (UGC). User generated content continues to grow. Facebook has over 500 million users populating pages with rich social meta data. There are over 300 million blogs. Wikipedia has more than 15 million articles. Content sources will propagate at geometric rates, especially as BRIC (Brazil, Russia, India, and China) countries up their adoption.
Proliferation of social meta data. Organizations seeking a marketing edge must digest, interpret, and asses large volumes of meta data from sources such as Facebook Open Graph. Successful identification of social graphs require matching gargantuan volumes of meta data (e.g. likes, check-ins, groups, etc) through introspection across a vast array of objects. Human centric and object centric events will inevitably coexist and engulf unified activity streams.

Figure 1. Activity Streams Improve Collaboration And Deliver Dynamic User Experiences