“But it wasn’t because I didn’t know enough. I just knew too much”. Gnarls Barkley – Crazy

The market is volatile. Your customers are volatile. Maybe if you look in the mirror you will see a volatile person. But volatility is something we – in general - tend to dislike. Therefore we strive to regain control. Control of our finances, of our customer even of our future. Management Information seems to help us with this. But the way we use management information or business intelligence is not always up to par. Most of the time we use historical data or performance to make decisions. But many answers, smart moves, right decisions are hidden in the near future. That’s why we want to be proactive. Beat the other guys. Advanced Analytics seems to work. It combines techniques like datamining and statistics with business knowledge to create predictive models. But even in advanced analytics we seems to focus too much on using our own data. Some argue that the most valuable information is not found inside your company but outside. If this is true I don’t know. What I do know is that external data can help you improve your performance. But how do you regain control of external – often unstructured – data. And even better. How can you make this process repeatable? The most obvious three problems that arise are: [1] too much data, [2] data too late and [3] poor data quality.

The problem with having too much data is that you need to come to a selection. The data in the world keeps growing and growing in mind boggling amounts. But the majority of this increase is not relevant. Pictures on flickr are great but aren’t much help when figuring out your supply chain strategy for 2010. But it is not so much the amount of data that is the problem. It is the structure. If you find a way to effectively structure any increase can be handled without too much effort. Therefore I would suggest to start structuring your data for relevance.

Another problem is that the relevant structured data is often received too late. After the fact has already happened. This means that no proactive behavior but reactive behavior. So how can we improve on that? One solution could be to set up early warning signs. Maybe there are some early indicators that can warn us about things to come. For more information on this I would like to suggest the work of Ben Gilad (http://knol.google.com/k/ben-gilad/competitive-intelligence/1o41pnd9hgmyg/2#). Please make sure you know what early warning systems are crucial for your organization.

Finally, good old data quality. In our pursuit for perfect data we tend to forget the reason why? Do we really need perfect flawless data ? If we have one piece of the puzzle that is wrong it might causes a problem. But if we have 1000 pieces of the puzzle – some right some wrong – we have enough information to make our decisions. So forget about data quality and start focusing on relationships between the data elements that make for the big picture.

If business intelligence can improve by adding external data, why is it that a lot of companies keep looking inward for their management information? One explanation could be that compliance is often the reason why they have a BI environment anyway. For some reason they have to report their performance to stakeholders. This mandatory reporting is often focus on (financial) internal data. Or  somebody started with improving a process by making it measurable with the use of BI. Or even more simplw. It is easier to start with your own system, company or IT department.

In conclusion I would like to argue that we need to structure and combine internal and external data in order to really gain any competitive value.

Times they are changing. Last year has been tumultuous for BI at best. We have seen the BI market consolidate into a few remaining mega vendors. But there was also room for niche players especially the ones focusing on speed, visualization or collaboration. So what will 2008 bring? To quote from Star Wars: “Impossible to see the future is”. But let’s give a try anyway. There are five mega trends that I foresee. Some have been around for quite some time and some are new. The first one is called: (1) Increase: Business Intelligence must be able to cope with considerable increase in both data (size) and speed (real time). The second one is (2) Integration: Business Intelligence will continue to integrate with operational processes but will also be more aligned with strategy. The integration of (un)structured data will increase. Finally, Business Intelligence will integrate with other business applications or ERP systems. Third, and my personal favorite, is (3) Insight: Business Intelligence will continue to become more intuitive (less end user involvement) showing an increase in advanced visualizations but also more analytic (more end user involvement).The fourth is (4) Interactive: Business Intelligence will be more reciprocal by nature; creating a dialogue between system and end users as well as sharing information and collaborating within performance networks. Finally, number five will be (5) Industrialization: Business Intelligence development will be characterized by a more industrialized approach using standard development frameworks opening up the possibility to outsource.  Increase will be mostly characterized by the need for real time information (partly due to the fact that BI becomes more operational). This will be supported by tools like Netezza and Teradata. They can also support the growing process of data retrieval from vast amounts of data. Integration of unstructured data (text, search, documents, agents) for example in a document warehouse (using tags) sitting next to the data warehouse will be the wave of the future. I expect Google or Attunity to take the lead here but the original BI vendors (Cognos, BusinessObjects) have also been investing in this. Although the market is more or less divided among a few remaining vendors it is still possible that we will see some more integration. This will be partly triggered by the fact that BI will be used on all levels: strategic, tactical and operational. This means BI tooling must support strategy development as well as pervasive BI (it’s everywhere). The renewed interest in open source Business Intelligence (Pentaho, Jaspersoft) is a clear counter reaction against the market domination due to vendor consolidation. Creating Insight today will be the one mega trend where I expect the most innovation to come from. It is a hot topic and fun to watch evolve. Here you will find things like intuitive intelligence (Strategy companion) or decision engines supporting your daily work (Fair Isaac). Also advanced visualization will help the end user get more grip (fractal edge) on things. Triggered by the success of “competing on analytics” or “super crunchers” we will see an rise in the use of analytics, strongly combining informatics, statistics and business knowledge. Perhaps here lies the opportunity for the long awaited promise of data mining? A clear and much appraised trend is the sharing of information with other people either inside or outside your own company to create mutual benefit. But it is more than just sharing the information it is also about sharing the reasoning behind it. How have we come to this conclusion and do you agree? It is asking feedback or contributions to your problem solving by your stake holders. All the characteristics of the web 2.0 can also apply here. This can also be done by using a trusted third party to solve your BI wishes by providing BI as a service. As BI is getting more mature by the minute I expect that we will see a standardization of best practices resulting in an industrialized approach in (standard) BI development. BI factories, often based in typical outsourcing countries, can provide high quality, low cost ready-to-use BI solutions. Some people say that after all the consolidating this will be a boring year for Business Intelligence. But if I look at these five mega trends I can’t wait. Happy new Year.   

Almost all human beings, except for the blind or visually challenged, are by nature visually orientated. Our eyes constantly monitor our surroundings searching for things that are out of the ordinary. This real time interface between eyes and the visual cortex of the brain has helped the human race survive by quickly recognizing danger. The great thing about this is that we are also able to filter out all the noise. For example, if you want to mail a letter and during the trip from your home to the mailbox you see every little detail, you probably never get there. So we ignore the stuff that we already know or looks the same all of the time. Only the things that look different get our attention. At the present time this ability to quickly spot dangerous animals or other threats is most often used for other purposes: shopping! Store owners, website builders, they all try to grab our attention using visual stimuli. Recently I was at the new Starbucks coffee shop at Schiphol airport in the Netherlands. My eyes quickly spotted a new Ready To-Go coffee tumbler. It was different from the normal ones because it had some kind of holiday season appearance. Being a collector of Starbuck Coffee Tumbler (I know this is a bit geeky…) I bought it on the spot. For Business Intelligence it is not so much different. As we are confronted with an enormity of data we try to find ways to interpret it. We try to find the information or even knowledge in this overload of data. Therefore we use Business Intelligence. But this information is presented to us in more or less the same manner, shape or form. Therefore we are quickly bored with it. Knowing this and the fact that our eyes and brains have this great ability to recognize patterns and trends we need effective visual communication of data that changes its shape or form to grab our attention when needed and allows us to quickly asses if the information presented holds some kind of pattern of interest. We need visual business intelligence that makes use of our intuitive skills. Just take a look at the work of Stephen Few (http://www.perceptualedge.com/) one of the evangelists of visual Business Intelligence or tools like Gapminder (http://www.gapminder.org/) to see what I mean.

I get most of my creative ideas when I have the time to contemplate. Often this happens under the shower or in the gym. I always thought the main reason for this was that since I had nowhere to go physically my mind could wander instead. I turns out that I am right, but only partially.  Recent studies show that the emotional state of people highly influence the way they process information. If you feel happy you are more receptive for outside stimuli. You start to see the beauty in things. Also there is an increase in creativity when you are in a good mood. The other way around, depressed people tend to focus on themselves and their problems. So now you know: I like taking showers.  Design can make a product more attractive. Attractive things create positive energy and state of mind. When asked to choose between products that basically do the same - have the same performance, price and so on – people tend to choose the attractive one. The Apple I phone is probably one of the best examples. So if attractive things work better (or at least create the feeling that they do) and stimulate creativity why is it that a lot of Business Intelligence solutions still look like spreadsheets, matrices or boring lists? I always encourage my colleagues to make their solutions super fancy sexy. Meaning that it should not only hold the correct data but at the same time it has to look good too. By presenting the information in a creative, nice looking way you increase the acceptance and usage of the product.  For Business Intelligence solutions the presentation and visualization are just as important as the content. Do you agree?

“Who loves not wine, women and song; remains a fool whole life long”. This is an old German expression which means having fun. I wrote about “weib” yesterday (Cybersex and Business Intelligence). So today I will focus on the “weib und gesang”.  Regular readers of my blog know that I am very enthusiastic about the upcoming trend in decision making to rely more on statistical analysis. The umbrella term for this is called analytics. Recent books about this topic are: Competing on analytics by Thomas Davenport and Super Crunchers by Ian Ayres (www.supercrunchers.com). You are no longer a nerd if you think that regression is cool again. For the non mathematical persons: Regression is a statistical procedure that takes raw data and estimates how various causal factors influence a single variable of interest. Ayres describes in Super Crunchers how analyzing massive databases can provide greater insights in human behavior. One of his case studies involves the work of Oren Ashenfelter. Oren can assess the quality of Bordeaux wines based upon regression (basic variables are: rainfall & temperature). There is much skepticism about his work among wine connoisseurs. Robert Parker – probably one of the world most influential wine writers - said: “it is really a Neanderthal way of looking at wine”. However it turns about that Ashenfelter predictions are much more reliable than the old fashioned way. The week from 5-12 October was democracy week. Over forty broadcasting companies worldwide paid attention to this on radio and TV. Here in the Netherlands we had a mock competition for a new national anthem. The current one which is called The Williams song – or Wilhelmus – is considered to be the oldest one in the world. You can hear the competing songs on this website: http://www.wijzijndebaas.nl/page/liederen. As a follow up to this program there was a segment on a company called Platinum Blue. They have come up with a tool that is called Music X-Ray. This is a complex computer program that breaks songs down into 30 or so component parts’ including rhythm, melody, harmony, beat, cadence, timbre, pitch and gives each a number. All hit songs fit the same pattern. If your song matches the pattern this increases the change of success dramatically. They claim that human decision making (does it sound like a hit, can we promote it like a hit) delivers a 10% success rate. However if you add a third factor – statistical analysis – this score goes up to 75%. You can see their presentation on youtube: http://www.youtube.com/watch?v=8lUewWlLgLo. Just the same as with the wine there is a lot of criticism. The X-factor judge Louis Walsh said: “The only technology I would use to pick a hit is my ears. I would not listen to a computer,” The pattern is the same in both cases. The traditional or old fashioned way of doing business is challenged by a new approach based upon statistical analysis leading to much skepticism. In my view this discussion is too much binary digit, true or false (I wanted to say black-and-white but I am still a technology consultant). In both of the above mentioned cases it is just a tool or technique - a guide to making the final decision, which always comes down to a human being. It is not about replacing the human decision making process with statistical analysis. But on the other hand there is enough evidence to support the fact that adding analytics to the decision making process leads to increased results. This can be done by creating a dialogue between Business Intelligence tools and their users, instead of a one-sided approach of data delivery only, leaving the decision making process to the end user. This going back and forth between user and tool can lead to something that I call intuitive intelligence: leading to insight without (much) thinking.