MDM seems like “deja vu all over again.” We’ve been fighting with metadata management, data accuracy and multiple data versions for years. Whenever MDM technology achieves a new milestone, the data problem intensifies. As you’ve pointed out, the increase in social media and unstructured content intensifies the challenge – and opportunity.

Big data and information overload

The problem with most MDM solutions (and their predecessors), in my opinion, is the fundamental premise that an explicit metadata structure can be developed and maintained without significant diminishing returns. (And, that any 80/20 rule that makes sense today will make sense in the future.)

At the same time as many organizations attempt to effectively classify data on the onslaught of even more data – we seem to find unexpected insight in the social media wild west. Mathias has a point about crowdsourcing. You’re right, it’s not the full solution to solve this MDM dilemma, but it’s on the right trajectory.

Semantic, Linked Data, Machine Learning and MDM core vs. context

The reason that we have key performance indicators is that there is a lot of data that is not relevant to decision-making. The diminishing returns of MDM is spending the time classifying data that is context to the business. Especially when there could be perfectly good semantic taxonomies out there (or available in the future) that enables finding patterns combining internal and external data.

Internal metadata structures can be rigid following what the organization thinks is the logical pattern of things. Data relationships are explicit. Patterns outside of these relationships can be obscured. Not so with semantic technology. Data relationships well beyond the traditional limitations of relational databases emerge. Unstructured data becomes structured. External linked data could be leveraged to reduce the internal MDM burden.

That’s not to say that MDM is dead. Organizations should recognize core processes that require MDM efforts. The emerging semantic market, much of it crowdsourced, can provide sanity checks. Machine learning and other techniques can reduce the burden on the 80% of data that is context. (Possibly highlighting what ought to be core.) It may becoming a kind of semantic outsourcing for non core functions.

Perhaps semantic is about the cross the chasm. It appears to be the technology that can provide a solution to the MDM problems organizations face. As always with emerging technology, there are scalability concerns and many solutions remain toolkits or industry focused.

OTOH, if you are in a fragile situation where a negative trend on product launch could kill your business, or for that matter, a better than expected market uptake could leave you with demand but no inventory or ability to satisfy that demand, more accurate information could be crucial. As with everything it seems, “it depends”…

I’m seeing this “good enough” test manifest itself in more and more IT investments. Some situations warrant exhaustive governance, accuracy, data integrity, and structure – and others are best serviced with a fast, agile, self-serve, opportunistic, ad-hoc, and “close enough” solution. The latter case seems to be generally when the data (or the underlying problem or opportunity the data is supposed to help address) has a shelf life or half life.

Cheers,
Matthias
@matzeller