Earlier this year I was interviewd by a journalist from www.ttm.nl about Business Intelligence and Logistics (Transport).

Follow the link for the pdf of the article (in Dutch).

Article BI Transport Logistics (Dutch)

One to think about this weekend. Does BI deliver on its promises or is the solution always in the next version?

Everybody deals with data all day long, from the board room down to the order entry department. Most of the time we do this without even thinking about the quality of the data. But when we do think about Data Quality (DQ) it often already has become an issue. Therefore our approach towards DQ is often negative (Garbage In, Garbage out). And also our approach is often technical (Field X in Interface Y is alphanumeric with 9 positions instead of numeric 8). So what happens? The IT department hurries towards the business owner with their negative technical problem often followed with an even more technical solution. Nine out of ten times this involves a tool from some vendor that promises a magic data laundry machine that cleans your dirty data. Let me tell you, there are not many business users that become enthusiastic after a negative technical speech. A more smart approach would be to make a simple business case. Bad DQ costs money, therefore good DQ makes money (or at least saves you some). That should be simple enough for any business user to understand.  As BI becomes more operational (do I dare to say real time) the chance that the data is polluted (or not yet cleaned in the ETL process) increases. Another trend is the combination of structured and unstructured data. Combination means coupling of two or more entities, also know as data integration. This is again an area where the DQ is crucial. In order to deal with DQ under these circumstances there are two options. Option 1: increase the performance by adding more of everything (CPU, memory, tuning IO and so on). Option 2: increase the performance by doing things smarter. This means creating some kind of solution that involves metadata or business rules. Also using a profiler tool to filter out the usual suspects is a smart way to go. Whatever you do, try to take a proactive approach instead of solving it after the ‘laundry is already dirty’. What can be the result of bad DQ? It can lead toward making the wrong decision (when based on incorrect data), it can cost money from an operational point of view (double mailings) or even lead to damage to the corporate image. It is safe to conclude that DQ is important for everyone. But by making it every ones responsibility it at the same time becomes no ones responsibility. There is always someone else that owns the problem, needs to fund the solution and so on. An at the end of the day still nothing happens to solve the DQ issues, leading to wrong decisions and more waste of money.  Let me explain what I mean using a real example. Department A has a customer database where customer A is known as Mr. A. Department B also has a customer database where customer A is know as Mr. B. During the data integration process Mr.A. and Mr.B. are seen as two different persons while in fact they are not. Should department A fix this? Why should they as this registration in their database of Mr.A. suits the purpose. The same applies for department B. Should the IT department fix it during data integration? If so, do they choose Mr.A. or Mr.B. in their Master Data approach? And by the way, who is going to fund this? It is not a problem for IT so they are not going to open their wallet. It is not a problem for department A or department B so they are not going to fund it. We have talked about the data delivery and the data integration, now enter the data subscriber. Department C uses the central data warehouse for their management decisions, corporate mailings and so on. They have problem with this whole Mr.A. and Mr.B. stuff but do not consider it their fault. So why should they pay for fixing this? This is what I mean. The whole DQ (and metadata issue for that matter) problem keeps being pushed from one desk to another and no one is going to solve it.  The central questions are: 1. Who owns the problem? 2. Who is going to solve it? 3. Who is going to pay for it? All the IT department can do is to identify the problem and facilitate the solution.  I would love to hear what you think about this. Please let me know and write your comments. Perhaps together we can find a suitable solution. In the meantime we keep our fingers crossed and hope for the magic laundry machine.

In school I participated in a Poem contest. Although I didn’t win I found writing poetry to be fun. But as with many fun things you forget how important they are and after time you do not practice them at all. Instead we find other not so funny things more important, like work. Well, I found a great new way to combine both work and poetry on TedC blog: http://datadoodle.com/2008/07/28/bi-haiku-from-the-uk/ TedC started with combining the ancient Japanese form of poetry called Haiku with Business Intelligence. A haiku poem (ideally) consists of 3 line of 5, 7 and 5 syllables. I can’t wait to see your contributions. Here is one example (which does not 100% follows the rules of a Haiku – but rules are there to be broken – right?). Extract from your systems.

Leave overnight to churn.

Predict the future

2008 has been quite a year for Business Intelligence. We have seen large platform vendors buying well known BI companies. The result is that the BI market will be dominated by three maybe four players. This market, some predict, will grow with another 10% this coming year. Some analysts have stated that small independent BI software vendors will profit most from this all. They will grow faster and will be more innovative than their large competitors. It can be expected that this will give way to a renewed interest in Open Source Business Intelligence (OSBI). OSBI vendors are able to present themselves as an alternative. One of their strong arguments is that they can so help prevent a vendor lock in. Another aspect is status. Early adaptors can make good cheer with their choice for OSBI. They consider themselves winners as they have chosen to step outside the trampled down roads. Finally, this also fits in the increased interest for open source software. But are the OSBI vendors ready for the fight? Can they compete with the four mega vendors? To do so they need to have a complete BI solution (breadth) or a terrific niche solution (depth). One of the competing differentiators is the claim that OSBI is cheaper. By making the code public the vendor can lower their cost for research and development. At the same time it creates a form of shared ownership. Another advantage is that OSBI vendors do not charge you for the software (licenses). Their profit model is based on delivery of services and support. The bottom line is that the total cost of ownership (TCO) for OSBI is often lower. But the game is not won on costs alone. The fit between the user requirements and the BI solution will be just as much a deciding factor. The general impression in the market is that the functionality of OSBI runs behind. They offer some nice (additional) solutions for reporting and analysis but that’s it. They are no real alternative for their commercial counterparts. But is this true? Let’s take a short look at what the OSBI vendors came up with last year. Pentaho brought BI to the iPhone, added a meta data layer to their BI suite v1.6 and made their product compatible with Sun Solaris 10 (two OS products, 1 solution, low TCO),  Also Pentaho and Ingres have agreed on a strategic partnership for OSBI. This means shared sales and marketing activities but also an integration of their products (Pentaho is certified for the Ingres Dbase).  Pentaho also agreed on a partnership with Infobright for data integration and ETL for all MySQL customers. They have also been actively expanding in Europe. Jaspersoft came with v3.0 which includes interactive web 2.0 interfaces, drag and drop technology, a new metadata layer and improved security. They have also been cooperating with Microsoft to optimize their BI suite for windows and office. MsExcel can now fully be used as a front end tool for their JasperAnalysis data analysis server. Ingres – as mentioned before – closely work with Pentaho but also with salesforce.com offering a CRM SAAS product (Icebreaker). It is safe to conclude that OSBI vendors are strengthening their position with partnerships within the OS community and by adding new functionality or technology to their BI proposition. They are also rapidly closing the gap with the commercial BI vendors (who are often halting as they face product integration). If they can keep this pace they will have their BI products at the same level as the commercial vendors within three or four years. OSBI has the potential to become a serious alternative for the commercial solutions. (This article has been written together with Leo Cardinaals. Leo also works for Capgemini BI and is an active member of the open source community.).