Monday’s Musings: Why Next Gen Apps Must Improve Existing Activity Streams

Published on August 30, 2010 by R "Ray" Wang

Upcoming Data Deluge Threatens The Effectiveness Of Activity Streams

Activity streams, best popularized by consumer apps such as Facebook and Twitter, have emerged as the Web 2.0 visualization paradigm that addresses the massive flows of information users face (see Figure 1).  As a key element of the dynamic user experiences discussed in the 10 elements of social enterprise apps, activity streams epitomize how apps can deliver contextual and relevant information.  Unfortunately, what was seen as an elegant solution that brought people, data, applications, and information flow into a centralized real-time interface, now faces assault from the exponential growth in data and information sources.  In fact, most people can barely keep up with the information overload, let alone face the four forces of data deluge that will likely paralyze both collaboration and decision making (see Figure 2):

  1. Massive activity stream aggregation by enterprise apps. Every enterprise app seeking sexy social-ness plans one or more social networking feeds into their next release.  The mixing and mashing of personal and work related feeds will leave users confused about context and lower existing signal to noise ratios.  Yet, proliferation will continue as users seek to bring aggregated sources of information into one centralized feed.
  2. Explosive growth in the Internet of Things (IOT). Beyond just device to device communications, the web of objects, appliances, and living creatures through wired and wireless sensors, chips, and tags will drive most of the growth in the internet in the next 5 to 10 years.  With an estimated 100 billion net-enabled devices by 2020, these networks seek to discover activity patterns, predict outcomes, and monitor operational health.  The massive amounts of sensing data driven into systems will not only overwhelm users, but also handicap the performance of today’s data warehouses, analytics platforms, and applications.
  3. Flood of user generated content (UGC). User generated content continues to grow.  Facebook has over 500 million users populating pages with rich social meta data.  There are over 300 million blogs.  Wikipedia has more than 15 million articles.  Content sources will propagate at geometric rates, especially as BRIC (Brazil, Russia, India, and China) countries up their adoption.
  4. Proliferation of social meta data. Organizations seeking a marketing edge must digest, interpret, and asses large volumes of meta data from sources such as Facebook Open Graph.  Successful identification of social graphs require matching gargantuan volumes of meta data (e.g. likes, check-ins, groups, etc) through introspection across a vast array of objects.  Human centric and object centric events will inevitably coexist and engulf unified activity streams.

Figure 1.  Activity Streams Improve Collaboration And Deliver Dynamic User Experiences

Figure 2. The Four Forces Of Data Deluge

Filters Improve The Signal To Noise Ratios And Drive Relevance

Given the tall task of repairing the relevance of activity streams under the four forces of data deluge, users need better filtering tools from their existing solutions.  Today’s rudimentary filters remind users of the simple search engines from the early 1990’s.  Users must have filters with the sophistication to cut across the big data challenges.  Filters must span across mediums such as pages, books, notes, photos, videos, voice, and others.  Based on 23 user scenarios, the 5 major categories of filters should include:

  • People. Requests focus around people, their relationships, and formal and informal groupings.
  • Location. Physical location attributes include spatial coordinates, topology, environmental conditions, vertical position, and others.
  • Time and date. Time and date plays a key role in parsing out historical data, multiple chronological perspectives, and forecasting and simulation.
  • Events. Events serve as a mega filter by relating people, location, time and date, and purpose.
  • Topics. Topics represent a broader filter that represents a generic “other” category in filtering.

The Bottom Line: Users Need Greater Control Over Their Point Of View And Next Gen Apps Must Deliver

Filters alone will not provide enough firepower to put users back in control.  Users must easily self-manage filters.  Self-learning patterns should be identified by the system.  Text analytics, natural language processing, and complex sentiment algorithms will play a role.  User driven advanced filters should at a minimum include:

  • Saved filters. Users save and share with other users their library of filters.
  • Trending. Users apply layers of filters to correlate complex multi-dimensional patterns.
  • Simulations. Users proactively test out scenario plans with existing data.
  • Predictions. Users apply pattern recognition and trending to test hypotheses.

Your POV.

Buyers, do you need help understanding how activity streams can improve adoption and ROI.  Are you suffering from data deluge?   Sellers and vendors, want to test out your next generation product ideas?  You can post or send on to rwang0 at gmail dot com or r at softwareinsider dot org and we’ll keep your anonymity.

Please let us know if you need help with your next gen apps strategy efforts.  Here’s how we can help:

  • Providing contract negotiations and software licensing support
  • Evaluating SaaS/Cloud options
  • Assessing apps strategies (e.g. single instance, two-tier ERP, upgrade, custom dev, packaged deployments”
  • Designing end to end processes and systems
  • Comparing SaaS/Cloud integration strategies
  • Assisting with legacy ERP migration
  • Engaging in an SCRM strategy
  • Planning upgrades and migration
  • Performing vendor selection


Reprints can be purchased through the Software Insider brand.  To request official reprints in PDF format, please contact


Although we work closely with many mega software vendors, we want you to trust us.  For the full disclosure policy please refer here.

Copyright © 2010 R Wang and Insider Associates, LLC. All rights reserved.

  • Great article. filtering feeds and just getting the information that matters in apps is crucial. Without the right filters in place, so much time can be wasted 🙂

  • Rick – thanks for sharing the insights into how you are handling the issue of filtering “big data”. This is an excellent start. Can’t wait to see the product! – Ray

  • @Ray (and @Doug), you nailed it. At ThingWorx, we are taking a much broader view of the potential of activity streams than the traditional “micro blog” approach.

    In addition to filtering, a few other key considerations (which we’ve built into our stream engine in ThingWorx):

    1) Richer contextualization is essential to enable better filtering. Contextualization can be explicit (metatagging/annotations) or implicit (relationships, “type” of stream entry, etc.). Sometimes contextualization can be modeled a priori, other times you need to be able to infer/discover it.

    2) Aggregation is extremely critical. Aggregation can amplify and identify new events/streams that are of interest, perhaps more than the original stream entries. As with contextualization, aggregation can occur “inbound” (pre determined aggregates to monitor/calculate) or “ad hoc”.

    3) Temporality is something that most stream implementations are missing. By this I mean that relationships change over time. For example, this cargo container was filled with pharmaceuticals on a ship from Europe to NYC in May, but it’s filled with electronics on a train from a factory to Shanghai in July. It is essential to be able to snapshot/trace relationships over time so that streams can be properly contextualized.

    4) Extensibility and richness of streams is an area we’ve spent a lot of time on. Streams should be able to have data/transactions attached to them that provide extra “value” and context to the stream entries.

    Clearly performance is also a consideration, and complex or new problems/opportunities often require new technologies. We’ve chosen to leverage a non-traditional database model for stream modeling and storage for all of the reasons described above (I’m not a fan of the term NoSQL – you can give people SQL-like capabilities against almost any data store).

    I also (as you can guess by the company name, ThingWorx) heartily agree that the intersection of the social and the physical world will lead to a massive set of new “feeds” from which significant value can be extracted.

    The potential of the IoT will take likely shape first within enterprises (fewer technical/privacy/security concerns, plenty of existing sensors/system/data, lots of low hanging opportunities), but the demands for dealing with the data deluge are the same as you have outlined above.

    The other missing link is the way in which users will interact with and extract value from these streams. At the lowest level, keyword searching of content is often needed. Simple filtering with hashtags or user/follower relationships are another first tier approach that works, but we think more is needed, particularly when streams are enriched with data well beyond textual content. Also, sometimes the raw data itself isn’t of huge value, but aggregates/transformations are.

    To that end, we’ve taken the approach of what we call “SQUEAL” – which integrates search (keyword and faceted), query, and analysis.

    I see activity streams (perhaps combined with complex event processing), along with a semantic storage and retrieval engine, as providing the foundation for the next iteration of BI/operational intelligence.

    The days of prepared cubes, OLAP (which isn’t all that on-line) and reports should go the way of the dinosaur. Decisions will need to be made much closer to real time, and traditional approaches generally lead to the “we sucked yesterday” result – assumptions or analysis in a time frame removed from the time frame for actionability.

    In any case, fun topic, and I look forward to your continued analysis of it.

  • Doug – The list is great and weak signals is a big deal that most of us have not yet encountered. These filters need more heft if we can consume all this information. Thanks for sharing as always and if you ever want to guest post, let me know! – Ray

  • Karen – thanks for your comments! that’s the book I still need to read up on. Okay time to buy on amazon and stop procrastinating! We can’t consume all this information and data deluge is a killer. What do others think? – Ray

  • Ray, Great post! In reading it, I was reminded of Tom Davenport’s work on the Attention Economy, With so much information flow and the constant distractions and multi-tasking now required of us in our day-to-day jobs, how do we focus our attention on the things that are important and relevant, and avoid being diverted with low-impact, but time-consuming activities. The idea of filters and trends to help manage this information deluge is a key solution. Thanks!

  • Ray,

    Tradtional tools have become blunt instruments as you point out: “Today’s rudimentary filters remind users of the simple search engines from the early 1990′s.” The search engines of today are unable to keep up with the “interconnectiveness of all things” (Douglas Adams – Dirk Gently) This manifests itself in the artificial distinction between structured and unstructured data, the need to follow concept and activity streams, the lack on semantic understanding among tools and the problem of important “weak signals.”

    Stuctured & Unstructured Data
    Tools persist to focus on databases or “content”. Yet content, today, is very structured, typically in XML & it uses language and grammar. Organizations that leverage typical search engines miss the deep web of structured data. Insight often requires both types of data. The distinction is a technical one that should not be exposed to users any longer.

    Activity Streams
    Once again, you are on to something. Filtering and search technology suffers from the document metaphor where the full meaning is theoretically packaged in single container with external references/hyperlinks. Individual snippets of conversation in social media or even threads in e-mail/social media are insufficient to gain meaning.

    Semantic Understanding
    Most next generation apps have no semantic understanding. Some often have a closed internal focus (Master Data Management) but no relationship to the real world out there. Google can adapt the search algorithm, but without semantic understanding, it will not be fully effective for finding deeper content.

    Weak Signals
    The increase in data and tools that rank information by references or number of words can tell us a lot about the status quo. Companies need to see the weak signals to see new trends, opportunities, threats etc.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts