Research Report: Rethink Your Next Generation Business Intelligence Strategy

Published on June 28, 2010 by R "Ray" Wang

BI Solutions Must Address The Information Management Matrix

A confluence of changing business requirements and on-going vendor consolidation leads many organizations to rethink their business intelligence (BI) strategies.  Many buyers face decisions to move beyond departmental solutions or to stay with an integrated apps based BI solution.  Meanwhile, some buyers must decide whether or not an integrated platform provides the right balance between business impact and cost of technology.  Additionally, most organizations seek support for new data types and new deployment options.  As BI continues to evolve from fragmented and historical reporting to pervasive, predictive, and real-time decision support, an organization’s success increasingly depends on the support for a expanding information matrix of (see Figure 1):

  • New and traditional data types. A proliferation of data types from social, machine to machine, and mobile sources add new data types to traditional transactional data.  Examples include content, geo-spatial, hardware data points, location based, machine data, metrics, mobile, physical data points, process, RFID’s, search, sentiment, streaming data, social, text, and web.
  • Visualization and reporting paradigms.  Users expect more than the traditional charts, gauges, and dials.  Web 2.0 innovations show how Rich Internet Applications (RIA) through tools such as AJAX, Adobe Flash and Microsoft Silverlight can create interactive BI experiences.  New and old paradigms include ad-hoc query builders, business performance management (BPM) systems, dashboards, production reports, scorecards, and advanced visualizations.  New visualization types include matrix charts, network diagrams, bubble charts, tree maps, word trees, tag clouds, phrase nets, and others.
  • Approaches and styles. Analytical techniques continue to improve as data volumes explode.  New and traditional approaches include advanced analytics, business activity monitoring (BAM), BI workspace, decision support systems, low latency BI, meta data generated BI apps, non-modeled exploration and in-memory analytics, scenario analysis, and OLAP.
  • Deployment options. With data coming from so many different sources, users are seeking new deployment options.  Common solutions in the BI portfolio include BI appliances, BI in the Cloud, BI specific DBMS, Mobile BI, open source BI, on-premises packaged BI apps, private BI clouds, and SaaS based BI.

Figure 1.  The Information Management Matrix Drives Next Gen BI


Explosion In Semi-Structured And Unstructured Data Challenges Existing Solutions

Along with new business requirements, the old world of structured data must make way for a plethora of new data types in unstructured data.  More importantly, solutions must support a growing number of industry vertical standards in semi-structured data.  Unfortunately, no single vendor can support all the data types that fit into the following three categories (see Figure 2):

  • Structured data. Structured data remains the most understood type of data.  Traditional sources comprise of data in transactional systems such as ERP, CRM, SCM and other database management systems.  Solutions conduct analysis via OLAP and traditional apps centric and database centric BI systems.
  • Semi-structured data. Common examples include flat files in record format, RSS feeds, XML documents, and data in spreadsheets.  Industry-specific XML data standards and cross-enterprise data-exchange standards such as ACORD, EDI, HL7, NACHA, and SWIFT will continue to grow as BI goes vertical.  On the document print stream front, new systems should support ASP, Met code, and PCL.
  • Unstructured data. Sources include natural-language text from e-mail, blogs, SMS, social networking sites, text fields, audio, video, and images.  Unstructured data represents up to 80% of today’s data sources.  Enterprises are challenged in discovering and organizing unstructured data for the real-time delivery of information to the right user.

Figure 2.  Data Types Fall Into Three Main Categories


Next Gen BI Must Reflect Emerging And Evolving Business Requirements

As market hype increases and sellers (i.e. vendors) rush to meet new requirements, buyers (i.e. users) should focus on 10 emerging and evolving next gen requirements that span dynamic user experiences, business process focus, and community connectedness (see Figure 3):

  1. Support role based designs. Information enables actionable insight by role.  Dashboards, reports, alerts, notifications, and filtered activity streams must deliver relevant information at the right time for the right person.  BI tailors to the needs of a role and should be self-service.
  2. Deliver consistent experience across channels and deployment options. Information enforces consistency across deployment models, devices, and data sources.  Users create, read, use, delete, and share information in a consistent user experience.  BI delivers information any and everywhere.
  3. Enable contextual, timely, and relevant delivery of information. Information knows when to provide; what information to; which users; where it makes sense in a business process.  Users receive activity streams of information based on preferences.  BI understands relationship history and how to deliver to the right person at the right time with the right authority.
  4. Align with configurable and adaptive business processes. Information aligns with business processes.  Users easily modify processes to meet changing conditions.  BI provides user driven ad-hoc reporting and configuration.
  5. Facilitate outcome-focus and results-orientation. Information ties back to goals and instills accountability for results.  Users gain transparency and insight in data.  BI tracks key performance indicators (KPI’s) and related actionable key performance drivers (KPD’s) across an end to end process.
  6. Foster proactive, predictive, & actionable insight. Information identifies patterns and trends including intent and forecast projections.  Users respond with speed and accuracy to requests and apply new techniques such as key performance predictors (KPPs).   BI anticipates requests and supports decision making.
  7. Empower all types of stakeholders. Information and results shared with internal and external stakeholders.  Users establish relationships and participate in communities and social networks.  Solution opens up the system to new types of users, collaborators, networks, and communities.  BI must be easy to use by everyone making decisions, not just business analyst gurus.  Solutions should also take into consideration accessible outputs for the visually impaired.
  8. Provide pervasive and natural collaboration. Information shared and socialized with key stakeholders.  Users collaborate with internal and external stakeholders to review results, modify plans, add understanding, and communicate changes.  Collaboration should support decision making.  BI embeds knowledge worker skills into existing work flows.
  9. Engage self-learning and self-awareness. Information supports decision support, presence, and learning.  Users build a history of interactions that drive preferences and identify patterns.  BI applies presence and location information to offer relevant actions and remembers patterns and behaviors.
  10. Permit security, scalability, and safety. Information supports role base security, data masking, partitioning, archiving, disaster recovery, and other security requirements.  Users shielded from cumbersome security measures.  BI meets regulatory requirements and disaster recovery thresholds.

Figure 3.  Next Gen BI Requirements Touch On 10 Elements Of Social Enterprise Apps

Long Term BI Strategy Must Support A Multi-Disciplinary Integrated Approach

Organizations can expect BI solutions to be embedded in a growing number of business solutions.  With actionable insight a scarce resource, natural integration points with other information management and orthogonal disciplines will emerge:

  • Business process management. A strong linkage between BI and business process management will enable operational BI (OBI).  Organizations will transition from reactive to proactive process management.
  • Content management. Unstructured information tied back to BI will bring new dimensions to information management.  Organizations can expect improved informed decision making and richer context to analytical reports.
  • Data governance. Formulation of an organizational strategy and data stewardship methodology must align with the BI strategy.  Organizations must model business rules and controls that span cross-functional teams to improve the success of data quality and BI deployments.
  • Master data management. Master data management forms the foundation of successful BI engagements by acquiring, cleaning, distributing, organizing, and managing master data.  Organizations still lag in MDM deployments but must come up to speed to support enterprise BI requirements .
  • Social technologies.  BI’s reach into social apps and the social metadata layer bodes well for an industry just starting to explore social technologies.  Organizations need to associate a Twitter handle or social networking account with a customer account.

The Bottom Line For Buyers – Compare And Contrast BI Strategies To Match Business Impact And Technology Value

As with all IT solutions, organizations must balance out the business impact with the cost of technology (see Figure 4).   Given the three major vendor approaches to BI, organizations must rethink which approach best suits their near and long term strategy (see Figure 5).

  • Application and business process centric. These solutions take an application or business process centered approach based on applications such as ERP and CRM.  Optimization for apps data and processes best suits organizations when more than a majority of their business information resides in a few data sources.  Sample vendors include Epicor, Lawson, Microsoft Dynamics, Oracle and SAP Business Objects
  • Pure play best of breeds. These solutions provide deep capabilities in a subset of data types, visualization and reporting paradigms, BI approaches and styles, and deployment deployment options.  Pure play best of breeds deal well with multiple data sources and support data heterogeneity.  These solutions often address the data integration (DI) issues head on.  Sample vendors include Actuate, BIRST, Information Builders, MicroStrategy, myDials, PivotLink, Proferi, and QlikTech.
  • Strategic BI platforms. These solutions seek to build a comprehensive approach to supporting data types, visualization and reporting paradigms, and BI approaches and styles.  Organizations should conduct a thorough analysis of platform maturity in order to assess degrees of integration, architectural consistency, and solution completeness.   However, not all strategic BI platforms are created equal as some are more apps centric versus apps independent.  For example (i.e. in order of most apps centric to most apps independent), the range includes Microsoft, Oracle, SAP Business Objects,  IBM, and SAS Institute.

In addition, there are a number of enabling technologies required for BI to be successful such as:

  • Data integration (DI) and data quality (DQ). DI and DQ solutions enable the access to, transformation of, integration with, and delivery of the best version of truth to business systems.  BI efforts require successful DI and DQ strategies.   Sample vendors include Capscan, DataFlux, Datanomic, Experian QAS, IBM, Informatica, Inquera, Melissa Data, Omikron, Pitney Bowes, SAP BusinessObjects (FirstLogic), Satori, Talend, Trillium, and Uniserv.
  • Data warehouses (DW). DW’s primarily provide a place for consolidated data storage.  BI systems leverage DWs to retrieve, analyze, extract, transform, load, and manage data.  Sample vendors include Aster Data, Greenplum, HP Neoview, IBM, Microsoft (DATAllegr0), Netezza, Oracle, ParAccel, SAP Sybase IQ, Terradata, and Vertica.
  • Master data management (MDM). MDM helps organizations apply a technology solution to the acquisition, cleansing, distribution, organization, and management of master data to systems such as BI.  Sample vendors include DataFlux (A SaaS Company), IBM (including Initiate), Informatica (previously Siperian), Kalido, Oracle, and Talend.

Figure 4. Extent Of Business Impact And Cost Of Technology Delivery Drive Business Value

Figure 5.  Comparing Vendor BI Strategies: When and Why?


When evaluating next gen BI strategies, consider the following best practices in vendor selection:

  • Focus on enabling all types of users. Solutions should enable anyone from a front line employee to a power user to view reports, build ad-hoc reports, collaborate, and improve decision making.  Design for roles could eventually result in BI activity streams filtered by key information needs.  Change management should be budgeted for solution development, training, and ongoing education.
  • Support highly differentiated processes.  Take advantage of industry specific solutions that meet critical vertical requirements and reduce time to market.  Over time, expect BI to continue to specialize by verticals.  Certain industries will require different levels of information optimization.  For example an emergency dispatch, may prioritize optimization of certain data types over others or regulations may require a certain standard of delivery.
  • Stay flexible. Ensure the systems supports multi-channel heterogeneous data sources.  Do not stay dependent on ERP systems for primary sources of data.  Expect a rapidly changing business environment that rewards flexibility.
  • Avoid vendor lock-in to one set of technologies. Seek different approaches and styles depending on the business requirement.  Be open to best of breed solutions as needed.  Keep in mind, support of specialized scenarios or cross-enterprise requirements may result in the selection of best of breed.  Apps independent vendors will be more focused on supporting heterogeneity.
  • Expect to move beyond departmental. Organizations will want to replicate successful deployments in departments across the enterprise.  Choose technology approaches and styles that can scale across different business functions, data types, reporting paradigms, and deployment options.
  • Factor existing infrastructure. Consider when it makes sense to invest in or throw out existing systems.  Astute next gen enterprises find ways to simultaneously leverage existing investments and innovate.  Plan replacement projects in phases.
  • Apply selection tools.  Save time and use independent vendor selection short list and check list tools to map business requirements such as organization and team structure, business and process maturity, technology strategy, and technology solution ecosystem.

Your POV

Does your existing BI strategy support these next gen requirements?  What next gen BI capabilities would you like to see your BI vendor deliver on in the next release?  Is your organization ready for a next gen BI strategy?  Does your BI strategy account for new social requirements?  Add your comments to the discussion or send on to rwang0 at gmail dot com or r at softwareinsider dot org and we’ll keep your anonymity.

Please let us know if you need help with your next gen BI strategy, overall apps strategy, and contract negotiations projects.  Here’s how we can help:

  • Designing a next gen information management strategy
  • Providing contract negotiations and software licensing support
  • Demystifying software licensing
  • Assessing SaaS and cloud
  • Evaluating Cloud integration strategies
  • Assisting with legacy ERP migration
  • Planning upgrades and migration
  • Performing vendor selection
  • Renegotiating maintenance

Contributors

Research from this document came from conversations with 37 buyers (users) and the following sellers (vendors):

Actuate IBM Cognos Informatica
Information Builders myDials Oracle
QlikTech Proferi SAP Business Objects
SAS Institute

Reprints

Reprints can be purchased through the Software Insider brand or Constellation Research.  To request official reprints in PDF format, please contact r@softwareinsider.org.

Disclosure

Although we work closely with many mega software vendors, we want you to trust us.  IBM and Oracle are currently retainer clients of Constellation Research but not a client of Insider Associates, LLC.   At the time this report was written, Altimeter Group had conducted project work for Informatica, QlikTech, SAP Business Objects, and SAS Institute.  This report was independently produced.  For the full disclosure policy please refer here.

Copyright © 2010 R Wang and Insider Associates, LLC. All rights reserved.

  • Caitlyn – The right info to the right people at the right time. What do others think? – Ray

  • […] Excitement in putting Business Intelligence (BI) to work. A combination of pent up demand, SAP marketing of Business Objects, and early adopters of BW led to many interesting conversations about the future road map.  Users sought clarity on the future direction and for the most part received it around BEX support and future investments.  Many continued to wonder if SAP would clean up its master data management strategy and address the need for a stronger next generation BI platform. […]

  • Ray, this report is a reality check for Business Intelligence. You’ve done a nice job outlining something I
    strongly believe – a one-size-fits-all approach no longer fits evolving customer requirements.
    BI has expanded well beyond its original definition when the term was coined by Howard Dresner, and
    the old BI guard has contributed to the problem with homogenized on-premise BI platforms that layer
    on more and more complexity. No matter how hard they work to label it “innovation,” I agree, it’s more
    about vendor lock-in.
    The next gen requirements you outlined are consistent with what we hear as customers roll out modern
    BI strategies. They’re embracing next gen BI solutions that embody aspects like role-based designs,
    which I agree, are about configuration and user self-service, not a vendor-defined role. Business process
    alignment is also key, particularly the ease and speed at which the BI solution adapts to changing
    requirements. These and the other elements in your top 10 (Figure 3) are rock-solid guidance for
    organizations that are embarking on this process.
    This report sparks a bunch of related questions, such as: When do you require a BI platform, and why?
    Does dashboard lipstick really make something BI? I’ve started to tackle a few of these in my separate
    post and look forward to continuing this outstanding discussion about where BI is headed.
    Thanks for the reminder of something Peter Townsend said, “…don’t get fooled again.”
    Dyke Hensen
    SVP Product Strategy, PivotLink

    More discussion here http://blog.pivotlink.com/2010/07/rethinking-business-intelligence-strategies/

  • Shirish – Great questions. 1. Decision making is the point of BI, especially at the role level. Do your employees have the right info to take the next action? 2. You are right. We should have the next blog post on this? Any takers? – Ray

  • HiRay,

    Thank you for a very comprehensive blog on next gen BI. I have come across some very fundamental issues and would like to know your POV.
    First, Most BI solutions are designed to help decision making. Do we ever actually evaluate the quality of decisions taken, in presence (or absence) of BI?
    Second,
    Do we actually find out what information is requied to make a high quality decision?
    How much information from BI is actually used for decision making and which other information has enhanced the quality of decisions?

    May be it calls for an independant blog.

    Thanks,
    Shirish

  • Zoe – thanks. that sounds like a good idea to build scorecards and dashboards tied back to relevant groups. as BI gets collaborative, you can expect solutions to allow you to publish and share reports with team members, groups with similar security settings. Have you tried Power Pivot with Microsoft? 100M rows in XL – sort on the fly =) – Ray

  • Smashing article.

    I have been working on BI project for a while at a company that uses a “big vendor” ERP. Our initial approach was very role based but what we are missing is something similar to what Doug was talking about. Role and Goal.

    So here is what we are doing now. Instead of Roles (that match the ERP) we build groups that are separate. Each group gets a set of metrics and goals, something that looks a bit like the balanced scorecards of the past.

    Currently we are using Microsoft BI stack (with some problems) and are going to add to that some open source in-memory DB cubes. I did notice that MS is not on your vendor list. Is because, like we found out, they are coming up way short?

    Again, good read!!

  • Wayne – good points. The intersection of analytics and scenario analysis could be called out more under the Business Process Focus if “Foster proactive, predictive and actionable insight”. This is the holy grail when looking for ROI in BI – Ray

  • Doug – once again, many great points. 1. Role and goal – based for some reason keeps getting the short end of the stick as people put lipstick on pigs (bad UI and call it role based). There needs to be a complete rethinking of the type of information needed to support decision making. Activity streams could be one approach, dashboards, alerts, notes, these are all elements that are required. 2. DIY BI – the holy grail. enough said on my end. 3. Great points on data and relationships. Add hierarchies and you have a full complement. Once again, great stuff on your end! – Ray

  • Ray,

    There is no question that social networking is showing us the BI path: role and goal-based design and DIY/self-service. It’s also showing how to break the structured/unstructured hegemony.

    1) Role and goal-based

    Enterprise software continues to suffer from “functional” design where users have to know where they are and where they are going. There hasn’t been a lot of advancement in the basic metaphor or dealing with screens as data input forms since the green screen days – just enhancing the form. Yet, social networking sites have been able to break out role-based designs that generally require little or no help. Finding and paying for an airline flight online is much easier than filling out an expense report in enterprise software.

    2) DIY BI
    Business intelligence software has always relied on the IT gurus who understand the SQL black magic and the underlying business rules of enterprise software to generate ad-hoc reports and cubes. It’s almost as if there are master data management trolls secretly transforming data in enterprise dungeons. We need DIY to enable data discovery and analysis through different visualization media – as you’ve described. Text reports are not that useful for decision-making. It is important that there are organizational KPIs that form the core of performance management, but there needs to be data discovery. The various BI categories are slowly merging and much of the capabilities are held natively in databases.

    3) Hegemony of structured and unstructured data

    This is going to be the next opportunity for innovation in business.
    -First, I think that we tend to discount the structured nature of unstructured data – there is often a structure of headers, footers, heading, bullets, subject lines etc.

    -Second, although structured data is well-understood, it isn’t as effective as many think. There are often relationships in data that is not stored structurally in a relational database. Addresses are an example: the address might refer to a city, state/province, country – but there are notions of west/north/east/south or Europe/Asia/Africa not in the database. (That’s why this need to be introduced as dimensions in OLAP). Title is another: a President & CEO in the United States is equivalent to a Managing Director in the UK.

    -Third, there are ways to integrate structured and unstructured data, probably through the use of semantic web. My sense is that this technology is crossing the chasm and is able to provide that extra insight that has been missing. In principle, we talk about drill down in OLAP from summary to detailed data – but what about drilling to documents? We may find that the relational database infrastructure that supports structured (SQL) and unstructured (blobs/XML) needs to undergo significant changes to be effective in the next generation BI. For one thing, MDM is likely to transform to an even more semantic exercise than it is now.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Related Posts