There is this guy, Bradford Cross, whom I met on my first project at ThoughtWorks.  I remember the day in a profound way, as I was on my first day at a client that, you could say, was something of a well known company in the top tier of accounting firms.  The kind of place one might perceive, without knowing further, was the kind of place that still may have a “business attire” dress code.  Well – here I am on day one, and this person walks in, I think wearing a green belt, unmatched pants, having messy hair – ready and eager to start work on this new project we were working on.

Welcome to ThoughtWorks, and Welcome to Silicon Valley!

Fast forward to today, and this same person, who I remember having a passion for functional programming and math that far exceeded anything I could muster up, is now one of the people behind FlightCaster – a service that uses functional programming (using Clojure and Hadoop) to implement a predictive algorithm that can tell you – with far more accuracy than the airlines – when your flight will arrive!  Being the cynical skeptic that I tend to be, I really did not believe it until I started using the service myself.  While I am only two flights in, the results are very promising.

To sum up what it does, the service uses 10 years worth of flight data, along with a certain amount of statistics wizardry that I almost certainly don’t understand, to correlate various events (plane arrival times, weather, etc.) to instances where delays occur.  More to the point, when I think of what business intelligence should be, what FlightCaster does seems far more interesting to me than what I have seen in most vendor presentations.  This is what leads me to the idea that BI comes from an idea about how correlation might occur, combined with technical folks who understand statistics and know how to harness data to do analytics.  It almost certainly never comes along because you install a tool.

So if that is where BI comes from, what is the most ideal toolset for helping to realize it.  There are no shortage of vendors who will try to sell you something that will lock you into their stack.  Or sell you some sort of whiz-bang UI that convinces you that you can go cheap on people so long as you just have good tools.  A typical anti-pattern is that an organization will buy the tools, then basically use them to do reporting against a data warehouse and post some simplistic results to some dashboard.  Something that, when you get down to it, you could have done without the tools, and probably without the fancy dashboard, at far lower cost using simpler tools.

This is especially true today.  We now have functional languages, equipped with great math and statistics libraries, that are for the most part mainstream (Clojure and F# come to mind).  We have open source tools like Hadoop (no more big expensive software) which make this stuff work in distributed computing environments that make it so you can leverage commodity hardware (no more big expensive iron).  If FlightCaster, a service that is as innovative as anything I have ever seen in the BI space, probably moreso, runs on this kind of stuff, it is probably good enough for what most companies would ever call their own BI efforts.

So what are the barriers to this?  One barrier I have encountered is that there is a bias against BI efforts that require “programming”.  I once gave a webinar about this topic to a group of CIOs, Consulting Executives, and BI product vendors.  One of my points was that the most useful BI efforts would require a technologist.  One guy in the crowed literally booed.  There is literally a subculture in the world of BI that thinks if you have to have someone program something, you have done a bad job.  Why this is the case I can’t be certain, but given that there are corners of the IT world that see programming as something to be avoided, it is not entirely surprising.

Another barrier is that there are more than a few CIOs or “Directors of BI” who are charged with the task of implementing a BI initiative, but have no clue where to start.  They know they need it, and that is as far as it goes.  Opportunistic salesperson comes on the scene, sells the SKU, and suddenly, we have a BI initiative made mostly out of shelfware.  It does not help that the world of “big database” (think IBM, Oracle, Microsoft) sells a database product that is largely a commodity without some proprietary BI extension which is usually added for differentiation purposes.  With a database software sales force already ensconced in the world of the enterprise CIO, it should suprise nobody that there are folks who think you need IBM, Oracle, Microsoft, or some other product to do BI.

I am here to say hogwash.  You don’t need any of that stuff.  The tools are here, they are free, and they have clearly done BI on a scale grander than most BI efforts I have seen inside most corporate IT.  What you need is a decent use case (tell me when my flight will arrive), that is solvable by a combination of data, math, and organizational will.  Do that, and you will have Business Intelligence.