Crazy BI
Writing by Jorgen on Thursday, 19 of February , 2009 at 9:56 pm
“But it wasn’t because I didn’t know enough. I just knew too much”. Gnarls Barkley – Crazy
The market is volatile. Your customers are volatile. Maybe if you look in the mirror you will see a volatile person. But volatility is something we – in general - tend to dislike. Therefore we strive to regain control. Control of our finances, of our customer even of our future. Management Information seems to help us with this. But the way we use management information or business intelligence is not always up to par. Most of the time we use historical data or performance to make decisions. But many answers, smart moves, right decisions are hidden in the near future. That’s why we want to be proactive. Beat the other guys. Advanced Analytics seems to work. It combines techniques like datamining and statistics with business knowledge to create predictive models. But even in advanced analytics we seems to focus too much on using our own data. Some argue that the most valuable information is not found inside your company but outside. If this is true I don’t know. What I do know is that external data can help you improve your performance. But how do you regain control of external – often unstructured – data. And even better. How can you make this process repeatable? The most obvious three problems that arise are: [1] too much data, [2] data too late and [3] poor data quality.
The problem with having too much data is that you need to come to a selection. The data in the world keeps growing and growing in mind boggling amounts. But the majority of this increase is not relevant. Pictures on flickr are great but aren’t much help when figuring out your supply chain strategy for 2010. But it is not so much the amount of data that is the problem. It is the structure. If you find a way to effectively structure any increase can be handled without too much effort. Therefore I would suggest to start structuring your data for relevance.
Another problem is that the relevant structured data is often received too late. After the fact has already happened. This means that no proactive behavior but reactive behavior. So how can we improve on that? One solution could be to set up early warning signs. Maybe there are some early indicators that can warn us about things to come. For more information on this I would like to suggest the work of Ben Gilad (http://knol.google.com/k/ben-gilad/competitive-intelligence/1o41pnd9hgmyg/2#). Please make sure you know what early warning systems are crucial for your organization.
Finally, good old data quality. In our pursuit for perfect data we tend to forget the reason why? Do we really need perfect flawless data ? If we have one piece of the puzzle that is wrong it might causes a problem. But if we have 1000 pieces of the puzzle – some right some wrong – we have enough information to make our decisions. So forget about data quality and start focusing on relationships between the data elements that make for the big picture.
If business intelligence can improve by adding external data, why is it that a lot of companies keep looking inward for their management information? One explanation could be that compliance is often the reason why they have a BI environment anyway. For some reason they have to report their performance to stakeholders. This mandatory reporting is often focus on (financial) internal data. Or somebody started with improving a process by making it measurable with the use of BI. Or even more simplw. It is easier to start with your own system, company or IT department.
In conclusion I would like to argue that we need to structure and combine internal and external data in order to really gain any competitive value.
Category: Analytics, BI Thoughts, Business Intelligence consulting, Business Intelligence strategy
- Add this post to
- Del.icio.us -
- Meneame -
- Digg
Comment by DaraghOBrien
Made Friday, 20 of February , 2009 at 12:21 pm
With regard to your point about data quality, I would have to disagree. What needs to be avoided is data quality for the sake of data quality, an obsessive compulsive desire for perfection.
However, you do need to understand the quality of your data (metrics, profiling, scorecarding) and have a capability to fix glaring errors that jump out of your BI (scrap and rework) as well as (hopefully) some form of plan to ensure that you don’t pollute your BI with crappy quality data.
“Quality” in this context can be defined as minimizes variation around a meaningful target (meaningful=specific consumer needs & expectations), not necessarily absolute perfection. If you have lots of data points to work with to validate the hypothesis your BI report is testing for, then the quality need not be perfect.
However, I still remember my experience as a call centre team leader many years ago when a BI analysis of lifestage data captured by a call centre team resulted in a large telco doing a massive marketing drive launching a new high-end internet service that actually resulted in old age pensioners being powerdialled by call centre teams to sell them a product they’d probably never use and probably couldn’t afford. That data was very wide of its “meaningful target”
Tom Redman writes about this type of issue in his latest book Data Driven.
Comment by SteveTuck
Made Friday, 20 of February , 2009 at 2:22 pm
Interesting post Jorgen, but I must pick you up on the issue of data quality.
If you are saying that you shouldn’t wait for your data to be perfect before using it in BI, then I agree; but to completely ignore the quality of the information you’re using to inform your decisions would be like playing roulette - Russian style. I’d also suggest that having too much data or data that is out of date are very much data quality issues.
No, you don’t need perfect data for BI, but you do need data that is fit for purpose and you therefore to be able to define what good data looks like and how you are going to measure (and if necessary) improve its quality.
These challenges also apply to external data, which all too often, imho, people see as a silver bullet. You need to understand the provenance of that information - where did it come from and when was it collected? And unless your’s and the external share a common key, you’re going to have to use some intelligent processing to integrate it in a way that will deliver value to the business.
As an industry we like to label things, but I see data management, data integration, data quality and data governance as different expressions of a desire to do the same thing - the creation, management and maintenance of data that is fit purpose for the business - all the purposes of the business.
Delivering on that requires processes, tools and technology that support all aspects of all the aforementioned list, but most importantly of all, it needs the recognition of the value of data to the business. For more of my views on this subject, visit my blog on www.datanomic.com.
Comment by andrewjbrooks
Made Friday, 20 of February , 2009 at 3:21 pm
Hi Jorgen, found out about your blog via a twitter comment…
It’s great to have views that trigger debate and learning, but I wondered if your advise to ‘forget about data quality’ was a typing error or a real point of view?
Nothing wrong with stating the good old ‘fitness for purpose’ argument (data needs to be sufficiently understood - it’s definition, lifecycle, etc. is accurate, complete, consistent, relevant, timely, trusted, etc. to the level required by the purpose to which data is being put, blah blah blah).
Nothing wrong in stating that, if you know all you need to know about data you are using to build your hypothesis or base your decision upon, including which bits are to be trusted and which bits are flawed (not fit for purpose), then you have the information you need to still make a ‘fact based’ judgement call.
However, if you are advising organisations to ignore data quality, you are advising them that it’s OK to have no understanding of whether or not some or all data being used to support a decision is ‘fit for purpose’.
I other words, unless your comment is a typo, you are suggesting most organisations should continue to ‘fly blind’ and not ‘Act on Fact’?
You must be logged in to post a comment.