There are many ways that a BI project can fail. And, many do just that. In my experience one of the top reasons is poor project management. Opinions as to why seem to fall into two camps. One side believes that BI is not inherently an IT project but is instead a business project and treating it like an IT project leads to failure. The other side opines that years of project management principles have been refined for software development and BI should learn from this.

Personally, I feel that both sides have their merits. I fully understand that there are some important differences with a BI project. And, there is no doubt that it must be driven by the business. Yet, I lean towards treating BI projects in a similar manner to software development project. In the future I hope to go into detail on how BI projects are different and hopefully will be able to provide advice on how to effectively manage these differences. In the meantime, what does it mean to treat a BI project like software and what exactly should be borrowed from the software development world?

Be Agile

To start with, the scope of a data warehouse and the ultimate business rules and requirements that it will encompass are vast. I contend that it is not possible to distill all of the business knowledge that is needed into a scope document; BI projects that try are bound to fail (at the very least they are susceptible to huge cost overruns and user adoption issues). With that thought in mind, a BI project does not lend itself to a traditional waterfall model.

Instead, a BI project should be agile and an iterative approach should be taken. There are several flavors of agile development methodologies and they are typically not geared towards a non-software project. So, lets look at some of the things that they do have in common: iterative development, a focus on team collaboration, and, perhaps most important, the ability to be adaptive to project change.

An Iterative Approach

It seems that many BI projects focus only on the big picture. The project can be on the small side and tasked with expanding a single subject area in an already existing warehouse. Or, it can be a huge build of an enterprise-wide data warehouse. Regardless, the focus often is on the pristine and perfect end product. I would argue that this is an enormous mistake.

There are a few reasons why I believe that this is a mistake. But, let me focus on the one. This style of development lacks transparency and creates a black box. It is difficult for the business to see the value in what is being developed. Instead, what they see is a lot of man hours being spent and they can quickly lose confidence in the project. This can be the death nail for a project and may very well hurt future BI efforts as well.

Alternatively, most BI projects can be broken into small components that take a short duration to complete. Each of these components can be prioritized and assigned to team members for a given iteration. Each successive iteration expands on the previous. So, after a single iteration you may not have a complete deliverable product but you do have something to show.

Take this example: you have a proposed large many-sourced product dimension that needs to be conformed. The work required to accomplish this task is greater than a typical development iteration. However, it can be broken down into even more fundamental parts. For instance, you might target a prioritized subset of the dimension attributes.

In the first iteration you’re able to have team members perform the business analysis work required to assess data quality, build a basic data model, and create the ETL to populate this data structure. In the end, you are not able to deliver a complete product. But, you are able to demonstrate progress. And, in each successive iteration you are able to build on this progress, learn from the work of the previous iteration, and ultimately deliver a complete conformed product dimension.

It seems simple enough. And, with a good team, it is. Each step of the way the business stakeholders have insight into what is being done. And, the likelihood of success is greater.

Still, there does needs to be some caution. A BI project is really just a small piece of the over all pie and should be part of a larger BI program within the organization. It is important to have the big picture in mind while working through the individual parts. You need to be conscious of and avoid building silos of data. The goal still needs to be to have a comprehensive and conformed view of the organization. It isn’t that agile doesn’t allow for this just be careful that goals do not get lost during the daily focus of an iteration.

Team Collaboration

Working with a team certainly isn’t something that is unique to agile development. But, there does seem to be a particular focus that shouldn’t be ignored. An effort should be made to have face-to-face interactions between the team members and it should be routine. Any roadblocks, as well as progress, should be brought to the teams attention as quickly as possible. And, attempts to involve business stakeholders should be a priority. After all, a BI project is for the benefit of the business and not just a pet project of IT, right?

Once again, much of this is about transparency. But, it is also about effectively working together and addressing problem early.

An entire book could be written on this subject and I can do it little justice in the limited space I have here. I simply encourage you to research the team methods used by the various agile flavors. And, please, do not overlook the importance of this. Good teamwork can make or break a project.

Adapt

A plague to many BI project (and software projects as well) are rapidly changing requirements. Many times the source of this is simply the result of a project digging deeper and deeper into the dark inner workings of a long running organization. Or, it could be the stakeholders changing their minds mid-stream. Whatever it is, it happens, it isn’t predicable, and you need to be prepared to deal with it.

The short iterative agile cycles are able to easily adapt to this change. While you will still have a high level project plan, required features, goals, and priorities the only thing that is really set in stone at any given moment is the current iteration. If a spec changes it is simply added to the list for future iteration developments. It is no longer an unforeseen showstopper. It is now just another requirement in the overall pool. In many ways this is a very liberating concept. Of course, there are still consequences to scope-creep. But, it is now an integral part of the process as opposed to a wall stopping progress.

Now What?

It goes without saying that agile methodologies are not perfect and that they will not solve all of the problems that a BI project will face. They are, however, flexible and able to provide a good base to build upon. As a part of a larger strategy they can provide important direction and proven guidelines. So, should you choose a specific agile methodology for your BI project?

As stated above, typically they are geared specifically towards software development. The flexibility, however, should allow you to adapt a methodology to a business intelligence focus. In fact, as an IT strategy it may be very possible to use a single methods across multiple project types (Scrum comes to mind). But, agile development methodologies are more about a mindset than a specific series of techniques. So, do not spend too much time dwelling on what some software pundits have turned into a religion.

The bottom line is that if you choose, and you should, to use a more agile development methodology in your BI projects you are going to need to adapt and tailor it to your organization. It is a mistake, however, to dismiss it as not being important or just the latest fad. Keep in mind that this is just a start. I will continue to expand on BI project management in future posts. If you’re not a believer now, just give me some time.