Why IT Matters More Than Ever

In 2003, Harvard Business Review published Nick Carr’s seminal essay, “IT Doesn’t Matter.” I remember that month well. The previous years of the PC boom, followed by the dotcom boom, had seen a tidal wave of money spent on technology. Much money was wasted on heavy investment in systems that either sat unused on a shelf, or ultimately were scrapped. To top it all off, the big “emergency” of the time, the millennium bug, turned out not to be as bad as everyone had feared. The time was rife for someone to declare the strategic irrelevancy of technology, and Mr. Carr stepped in and made the case. Because many executives desperately wanted to reduce technology spending at the time (in the aftermath of all that gluttony), reading Carr’s article was like receiving manna from heaven.

As a technology practitioner myself, however, it should surprise no one that I believe Mr. Carr to be wrong. Regardless of any biases I may have, the reality is that his argument contains several holes that have become ever more glaring with the passage of time. My intent with this article is to shine a light on what’s wrong with the “IT doesn’t matter” argument. I’ll even go further: I’d like to demonstrate why, now more than ever, investing—not just in technology, but in the intersection of technology and people—is the most important action that a company or organization can take.

The Company That Couldn’t Run a Second Shift

In my years as a technology consultant, I’ve certainly seen what happens when you don’t invest in technology. One client I vividly remember—we’ll call it Company X—had quite a conundrum. An amalgamation of nine different manufacturing organizations built up over time, Company X spent very conservatively on technology. When making an acquisition, Company X would do as little as possible on integration; for example, only enough to get the accounting systems and the order-entry systems sending orders to a central mainframe.

Over time, this management behavior at Company X resulted in a “Rube Goldberg” system requiring the work of hordes of COBOL programmers just to keep running. Overnight order processing required more than 200 manual steps. The maintenance cost of this Frankenstein’s monster of a codebase included at least 40 full-time COBOL programmers, a couple of dozen system operators, and who knows how much management hierarchy on top of all that. Company X was spending an estimated $15 million annually just to keep the system at its current level of functionality.

Unfortunately, that $15 million wasn’t even the most significant problem. When demand increased, Company X decided to run a second shift. The issue was that the system literally could not process orders in greater than about a 12-hour window. Moving to a 16-hour window, and eventually to a round-the-clock business, would require either radical changes to the old system or building an entirely new system—and it would have to match the quirks of the old system, in order to maintain compatibility with the nine other satellite systems!

Business analysts estimated that adding a second shift would increase gross margin in this low-margin business by around 3%, while increasing top-line revenue by over 20%. Moving to round-the-clock processing was also a precursor to being able to sell into new global markets, which would have increased top-line revenue even further.

Sadly, none of these happy improvements could come to pass for Company X.

The problems at this company were many, of which technology was merely one aspect. The deeper issue was that the operators and programmers who understood that mess of a codebase had a vested interest in keeping the current technology set running. But even if all those people were replaced, technical debt in Company X had built up over a long period of time, to the point that the company would need to invest around 2–3 times its annual profit in order to get out from under the load of technical debt.

In this culture of hubris, fear, and inertia, any new technology initiative to pay off the technical debt would almost certainly be killed. It was literally a recipe for company bankruptcy. When competitors are blowing you out of the water with 3% higher margins and 20% higher top-line revenue, taking that increased profitability and investing it into further productivity-enhancing technologies, it’s hard to see why “IT doesn’t matter.” If you’re a CEO in a company that can’t compete because of technical debt, you almost certainly will understand just how much IT does matter.

The rest of this article is available at Pearson Education’s InformIT website here.

Why IT Matters More Than Ever

The Root Causes of Technical Debt

When we spend 80% of a development budget just keeping software that we already presumably “own” working and current, we know that technical debt is extracting a terrible toll on our budget. When making a simple program change requires effort measured in weeks rather than days, something has gone horribly wrong. In many circles, the term custom software has essentially become synonymous with “blank check” and “will drain your budget for a long, long time.”

Do you champion change to help reduce such technical debt? You may eventually be rewarded for your foresight—but meanwhile the risks of championing change are very high. If you want to write maintainable software, a multitude of barriers stand in your way. When you admit that you can’t state with absolute certainty what will be delivered, do other people paint that disclosure as an admission that your team lacks discipline? Do your developers who have not yet tried pair programming or test-driven development reject those techniques without ever really attempting them—just because they’re new and initially uncomfortable?

Previous articles in this series have explored the problem of technical debt, the ultimate costs of this problem, and how methods that work only at surface levels stifle the possibility of changing to something better. In this article, we’ll consider why these problems affect our technology organizations in the first place—how technical risk factors into the technical debt that plagues the software industry.

Taking Down the Hubricists

I defined the term Hubricist in my article “Scrummerfall: World’s Worst Software Development Methodology.” The Hubricist attempts to impress those above him by replacing real data about the real velocity of the project with hubris. What the Hubricist lacks in substance, he makes up in bravado: “Yes, we will hit the deadline. Yes, you will get all the features!” When neither happens, developers are blamed for not working Sundays in addition to Saturdays, and careers are derailed in the aftermath.

Why is the Hubricist so opposed to change? For a moment, try to think from his point of view. Somebody (perhaps it’s you) comes along and trots out this methodology that promises transparency, particularly with regard to quality and technical debt. Is this methodology good for you, Mr. or Ms. Hubricist? Where would you fit into a regime that values results over bravado? Results-oriented management might not allow you to blame other people when things go wrong. If customers see that developers are making progress on a daily basis—working hard, but not at the unrealistic rate of progress you promised in order to sell the project, where does that leave you? Exposed, having made a lot of promises that can’t be kept.

With this kind of mindset, it should be no surprise that Hubricists try to block adoption of Agile methodology, or any other system that adds transparency. Remember the scene in The Wizard of Oz where the curtain draws back to expose the little man behind the booming voice? When the Hubricist perceives the potential for exposure, he plays the FUD card: fear, uncertainty, and doubt. He walks around claiming that Agile isn’t disciplined, that it’s about developers who don’t want to write documentation, that it’s advocated by teams that don’t want to be held accountable for results.

What can you do to counter the Hubricist’s claims?

  • Point out that Agile teams actually tend to produce far more useful documentation than other methodologies can. These are real showcases that demonstrate working software, versus mere status reports with vague claims of xx% complete, based on nothing but self-reported guesses.
  • Note that accountability for results must be based on a solid track record of delivery. Is a high percentage of the development budget spent on software maintenance costs? Past results don’t allow you to claim “accountability” for the future unless you have changed those statistics.
  • Show a real Agile team. Show stakeholders the daily artifacts of story cards moving from backlog to completion. An Agile team room is an active place, where collaboration tends to be visible and signs of progress are frequent. Compare this to a typical cube farm in a waterfall project.

The rest of this article can be read here.

The Root Causes of Technical Debt

Business Intelligence does not Come From a Product

There is this guy, Bradford Cross, whom I met on my first project at ThoughtWorks.  I remember the day in a profound way, as I was on my first day at a client that, you could say, was something of a well known company in the top tier of accounting firms.  The kind of place one might perceive, without knowing further, was the kind of place that still may have a “business attire” dress code.  Well – here I am on day one, and this person walks in, I think wearing a green belt, unmatched pants, having messy hair – ready and eager to start work on this new project we were working on.

Welcome to ThoughtWorks, and Welcome to Silicon Valley!

Fast forward to today, and this same person, who I remember having a passion for functional programming and math that far exceeded anything I could muster up, is now one of the people behind FlightCaster – a service that uses functional programming (using Clojure and Hadoop) to implement a predictive algorithm that can tell you – with far more accuracy than the airlines – when your flight will arrive!  Being the cynical skeptic that I tend to be, I really did not believe it until I started using the service myself.  While I am only two flights in, the results are very promising.

To sum up what it does, the service uses 10 years worth of flight data, along with a certain amount of statistics wizardry that I almost certainly don’t understand, to correlate various events (plane arrival times, weather, etc.) to instances where delays occur.  More to the point, when I think of what business intelligence should be, what FlightCaster does seems far more interesting to me than what I have seen in most vendor presentations.  This is what leads me to the idea that BI comes from an idea about how correlation might occur, combined with technical folks who understand statistics and know how to harness data to do analytics.  It almost certainly never comes along because you install a tool.

So if that is where BI comes from, what is the most ideal toolset for helping to realize it.  There are no shortage of vendors who will try to sell you something that will lock you into their stack.  Or sell you some sort of whiz-bang UI that convinces you that you can go cheap on people so long as you just have good tools.  A typical anti-pattern is that an organization will buy the tools, then basically use them to do reporting against a data warehouse and post some simplistic results to some dashboard.  Something that, when you get down to it, you could have done without the tools, and probably without the fancy dashboard, at far lower cost using simpler tools.

This is especially true today.  We now have functional languages, equipped with great math and statistics libraries, that are for the most part mainstream (Clojure and F# come to mind).  We have open source tools like Hadoop (no more big expensive software) which make this stuff work in distributed computing environments that make it so you can leverage commodity hardware (no more big expensive iron).  If FlightCaster, a service that is as innovative as anything I have ever seen in the BI space, probably moreso, runs on this kind of stuff, it is probably good enough for what most companies would ever call their own BI efforts.

So what are the barriers to this?  One barrier I have encountered is that there is a bias against BI efforts that require “programming”.  I once gave a webinar about this topic to a group of CIOs, Consulting Executives, and BI product vendors.  One of my points was that the most useful BI efforts would require a technologist.  One guy in the crowed literally booed.  There is literally a subculture in the world of BI that thinks if you have to have someone program something, you have done a bad job.  Why this is the case I can’t be certain, but given that there are corners of the IT world that see programming as something to be avoided, it is not entirely surprising.

Another barrier is that there are more than a few CIOs or “Directors of BI” who are charged with the task of implementing a BI initiative, but have no clue where to start.  They know they need it, and that is as far as it goes.  Opportunistic salesperson comes on the scene, sells the SKU, and suddenly, we have a BI initiative made mostly out of shelfware.  It does not help that the world of “big database” (think IBM, Oracle, Microsoft) sells a database product that is largely a commodity without some proprietary BI extension which is usually added for differentiation purposes.  With a database software sales force already ensconced in the world of the enterprise CIO, it should suprise nobody that there are folks who think you need IBM, Oracle, Microsoft, or some other product to do BI.

I am here to say hogwash.  You don’t need any of that stuff.  The tools are here, they are free, and they have clearly done BI on a scale grander than most BI efforts I have seen inside most corporate IT.  What you need is a decent use case (tell me when my flight will arrive), that is solvable by a combination of data, math, and organizational will.  Do that, and you will have Business Intelligence.

Business Intelligence does not Come From a Product

Some Reflections on Being “Nomadic” – 10 Tips for Road Warriors

It has been a long run, but after working in the following cities over the past 37 weeks:

  • 10 weeks in Seattle
  • 1 week in Los Angeles
  • 1 week in Las Vegas
  • 12 weeks in San Jose
  • 6 weeks in Beijing
  • 1 week in Orlando
  • 3 weeks in San Francisco
  • 3 weekends in places like Winnipeg and Minneapolis for Code Camps

… it appears as I will be back in Chicago – for at least a week! (I know… wow!).  The last one was a project we finished in 3 weeks out of 5 budgeted – which was a nice way to end what has been the most insane run of travel I have done in my career.  Now, I work in consulting, so I will likely be on the road again soon, but I thought I would take this opportunity to provide some thoughts on what works and what does not work when you are on the road as a traveling consultant.

Tip #1: When Possible, Get an Apartment

During my longer stints, in this year’s case, Seattle and San Jose, I managed to at least get corporate apartments that had fully supplied kitchens and laundry facilities.  While such arrangements are not practical for shorter projects (i.e. 2-3 weeks), I found that having such a place is a huge win/win for my clients as well as myself.  The cost of a corporate apartment is almost always lower than a hotel room – despite being far larger.  But more importantly, at least with me, is that I tend to prefer to cook my own food – which costs far less than spending a typical $35 per day on restaurants.  I often would have a net food cost – eating very high quality stuff, well under $100 per week, which would easily be blown on 3 meals during a week at a restaurant.

Probably more important though than the cost element is that I always felt healthier at the various corporate apartments.  I would routinely eat healthier when I prepared stuff myself.  It also kept my domestic rhythm going, as the practice of constantly eating out can make you stop remembering how to operate in a kitchen.  Same goes for having laundry facilities – something for me makes me feel normal if I have a daily routine, regardless of where I am, that involves taking care of myself.  Seems to make life, for me at least, feel more “normal” on the road.

Tip #2: Find the Local Bookstore (or Local “Whatever It Is You Like Doing”)

I am a bookworm – I read 3-4 nonfiction books a month, and probably skim through at least a dozen more.  Whenever I arrive in a new city, the first thing I do is find where the nearest good bookstore is, preferably an independent if I can find one, or the B&N or Borders if I can’t.  On the road, unless you are working 14 hour days (god help you), you want to find something to do with your idle time that isn’t work.

Tip #3: Do your Expenses Weekly

(note: I say this as I stare at a basket with 3 weeks of receipts in it…  ugh).

Expenses is a minor hassle if it is a weeks worth.  Much past that though and you a.) have what feels like an insurmountable burden in front of you and b.) you tend to forget things.  Just make it part of your routine to do them on Thursday night.

Tip #4: Avoid Too Many “Drinking Adventures” With The Team

This may be controversial – and thankfully, I learned this more in my twenties than in my far more tame late thirties I find myself in today, but man, does your effectiveness go down when you come into a client hung over.  I see people tend to drink more than they would at home when they are always going to restaurants, out with the team, and so forth.  After about 4-5 nights in a row of it, your liver will be screaming at you.  Do it, but certainly not to excess.

Tip #5: If You Have Family, Make Sure They Know You Still Exist

Another one where I all too frequently fall short, especially with west coast travel when I work too late and find myself home past bedtimes.  If you have kids, or a spouse, make sure they know you are around at least every couple days.  I swear, I got in the worst arguments with my spouse when I forget to call for a few days, and suddenly need to call because I need something “done at home”.  It is too easy to get lost in your project and forget to do things like that (at least for me)… but staying in frequent contact is critical if you have a family.

BTW, even if you don’t, tip #5 still applies to friends and extended family – though probably not at the same duration.

Tip #6: Stay Connected to your Tech Community

User groups mid-week in your home town are going to be hard to attend.  Make sure to use code camps or other weekend events to continue to stay connected and keep those networks intact.  One of the worst things you could do is take up travel, and then find that nobody remembers you 8 months later!

Tip #7: Use Travel to Expand Your Network

Find the local user group in the location where you are staying, and find a way to attend, or even better, contribute.  Travel, all things being equal, should be a net positive with regard to your network.

Tip #8: Airplane Time = Book Time, not Work Time or Internet Time

One of the worst developments I can possibly think of, at least for me personally, is the “GoGo Internet Service” that has been invading American Airlines planes.  Not that I am not thankful for innovation or what have you – but I use that time to take in materials that, well, take a long time to take in.  Whether it be books I have been wanting to read that require a good deal of concentration to successfully grok, or anything else that requires not being around distractions, leveraging airplane time to get those things done is one of the best ways to take advantage of that “temporary captivity”.

Tip #9: Spend at Least One Day On Location Doing “Touristy” Stuff

During my first four years as a road warrior, I visited places from Sydney to Madrid to New York and many places in-between.  Yet – I have never been to the Statue of Liberty, the Louvre, or other places I have perhaps walked by, but never really took in.  It took many years to remember that when being out here, I ought to use the opportunity to actually see a museum or do some of the things that, at the very least, give you interesting conversation fodder.  Getting to travel the world, and then promptly choosing to simply work on some 9th draft of a proposal in your room rather than experience some wonder of the world is, well dumb.  I know.  I have done it, and I still regret it!

Tip #10: Stay Healthy

Work out.  Eat healthy.  Read a balanced diet.  Don’t overwork.  Take care of yourself.  I know far too many people that burn out with this stuff because they forget that, even if you are on the road, you still have to take care of yourself.  Just because your employer is sending you somewhere does not mean they own you every hour of every day.  A burnt out consultant who goes through zombie like motions for a client isn’t terribly valuable to anyone, consulting firm, client, or themselves.

Happy Road-Warrioring!

Some Reflections on Being “Nomadic” – 10 Tips for Road Warriors

Don’t Enron Your Software Project

Yes, I decided to make “Enron” into a verb in my latest InformIT article on technical debt titled “Don’t Enron Your Software Project”.

The idea is quite simple – if you are hiding technical debt rather than disclosing it to the project sponsor at the time of turnover of version 1, you are really doing something similar to what the guys at Enron did – namely, selling an asset that has hidden, undisclosed debts attached to it.

Read more here.

Don’t Enron Your Software Project

Back from China

Sorry for lack of posting, but true to my moniker as a “Nomadic Developer”, I have been in China off and on (mostly on) for 5 weeks, working with a team on a project that we are doing at our ThoughtWorks office in Beijing.

Some thoughts so far. The first thought is how awestruck I am by the dynamism of China as a country. You always hear about it, but it is amazing to actually get to see it in person. Every time I go on a walk in Beijing, I am awestruck at just how vast of a city it is. Imagine something like the Chicago Loop, but it just keeps going and going, in all directions, for miles. That is basically what Beijing feels like to me.

However, the second thing I have observed – or really more just confirmed – is that China is really coming along as a place for software development talent. We all knew it would happen, I am just not sure I knew it would happen with such speed. During my trip, I paired with several developers from our office on our project, and truth be told, I am certain I learned far more from them then they could have possibly learned from me.

One example – and perhaps this is a ThoughtWorks thing generally – or maybe specific to China, is that on that project we are constantly pushing what we can do in our editor forward. Every time we repeat some action – such as finding usages for symbol – or anything else – rather than just muddle through using a mouse, we find a way to make it a keyboard shortcut. Pairing with this group has probably helped me increase my own developer productivity 2x, just in being around people who are constantly molding and navigating a codebase through ReSharper keyboard shortcuts (if you are not a programmer, ReSharper is a tool that adds a ton of productivity to Visual Studio, the typical tool people use to do .NET development).

For anyone who has some notion that “offshore” developers are somehow “lesser”, I can be sure that you can disabuse yourself of that notion immediately. I never learned so much in 5 weeks.

What does this mean for consulting in general? Well – I am becoming more and more convinced that, beyond the boutique stage of a consultancy (<10M in 1 or a small handful of cities working purely on founder relationships), if you have no global story, you are going to face an uphill battle. There is simply too much talent overseas, and the barriers to trade are simply too low, for you to be cost competitive on a large project. So much so that, if you were to ask me what skill you need to learn to keep your software development career humming, it would not be a programming language – it would be the Mandarin language.

Back from China

The Inner Hubricist in All of Us

The first article in my series about technical debt is up at InformIT, covering the concept of “Scrummerfall”, the insidious mixing of Scrum and Waterfall so you can pretend to be agile, but sadly, end up with simply a mess.

In the article, I introduce the persona of the “Hubricist”.  The Hubricist is the inner demon in all of us that just wants to seem “in-control”, and present an appearance that not only is everything “ok”, but that there is no risk, and progress on the project will be a nice, predictable, upward march of progress.

As much as I call out that persona in the article, recently, and even not so recently, I find this a persona that it is all too easy to fall to when dealing with the intersection of ambiguity and demanding customers.

In the best of all worlds, we can all be frank about how progress on a project is going. The problem is – even on a well intentioned team with good management support and everyone “on board” for doing “full agile” (whatever that means), you cannot escape the fact that we are all human.  No matter how agile you are, chances are, if you are human, you would rather not have confrontation than have it.  It is simply easier not to deliver a hard message – or sugar-coat one, than it is to deliver a hard message.  I believe the root of the inner Hubricist in all of us comes from the fact that we are human, and the Hubris of feigned confidence is simply a means of avoiding what is often a harder conversation than we want to have.  Lacking unlimited courage and “anything to lose”, it is inevitable that personal hubris on the part of those who report progress, reducing transparency.

So here is the reality – no matter who you are working with, there will always be a tendency toward Hubris.  The benefit of agile is not that it eliminates Hubris, but that it reduces the likelihood that it will be hidden.  If you are an on-site customer, chances are, you will have a better means to be a participant rather than an observer, which drastically increases the quality of information you receive.  You will see cards moving across the card wall directly.  If the team is healthy, you will constantly be seeing new bits of customer value demoed to you as they complete.

Better than that though, as an on-site customer, you have a chance to develop relationships with the team, which over time, increases the quality of communication.  As you gain a level of trust with the team, the team becomes more comfortable delivering both good and bad news.  To me, the very definition of a healthy consulting relationship is one where both customer and consultant are engaged in development of the solution on a day-to-day basis.  Avoiding Hubris requires courage, but it also requires trust, which is created through active collaboration.

As developers though – we have to constantly be on the lookout.  You have to become self-aware of when you are flying on hubris.  Success depends on habituating the practice of becoming real uncomfortable when making Enron style claims about progress.  It is tough, and sometimes, it feels risky, but in the end, replacing progress with Hubris is a devil’s bargain that only gives you – and your colleagues – more pain to be dealt with later.

The Inner Hubricist in All of Us

New Book Excerpt – Jargon from the World of Consulting

On InformIT.com, I have published a selection of jargon terms from my book.  Some of the terms many consultants will be very familar with, some perhaps not so.

I am also taking this opportunity to announce that the next article series on InformIT will be diving into the topic of “The Economics of Technical Debt”.  As an officially certified armchair economist*, it is my duty to offer up my own analogies between bad code and bad economics in this series that (3 articles in, soon to be published in short intervals) skewers the notion that you can write bad code and simply leave the consequences for someone else.  Especially in consulting, where presumably you are trying to gain clients for the long term (i.e. long enough such that you can’t just walk after leaving them with a mess).

New Book Excerpt – Jargon from the World of Consulting

In Consulting, Human Capital Matters More than Financial Capital, Part 2

In the previous post in this series, we explored the relationship between consulting and financial capital, and how the latter often is very harmful to the former, particularly in large amounts.

The real question, though, is what that implies.  Indeed, the chief reason given of why companies must always grow, is that:

  1. Capital seeks the highest possible rate of return
  2. Companies, who need financial capital, must compete for capital to expand
  3. Expansion raises the rate of return, based on economies of scale
  4. Higherrate of return attracts more capital
  5. Go back to step 1.

This is, of course, why you and I are not as rich as the heirs to the Walmart fortune.  Traditionally under capitalism, more capital meant you were more competitive, and you would grow, edging out all your competitors until either a.) the anti-trust police get you (from Standard Oil to AT&T to IBM to Microsoft) or b.) you somehow screw up and lose all your customers (GM, Chrysler, PanAm, Arthur Andersen).

Of course, if you consider the size of financial capital as only a minor factor, then you would want to look for other factors that control how a consulting firm differentiates itself.  How might such a firm do that?

Well, that depends on the kind of business you are.  The success of a technology consulting organization is going to depend chiefly on:

  • Relationship building capabilities of the consultants and demand generation teams
  • Brand reputation – being known for solving hard problems
  • Recruiting capabilities – ability to attract people who have both technical aptitude and relationship skills

So how does one increase those?  Especially if you can’t buy it directly?  Well – you have to become very good at attracting the best people, and once they are there, keeping them engaged.  If people matter most – you have to wonder how you attract and retain the people you want.

Can money buy people?  One approach to building a great consulting firm might be to offer to pay everyone $1 million us dollars per year.  People have done studies that indicate money does not, in fact, buy happiness.  If suspect what occurs in this scenario is you end up with a bunch of very rich, but very nervous, consultants who still will search for meaning in their work after a few months of “being rich”.  If anything, my experience with people who have sudden wealth is that they either become people you would rather not work with from sudden materialism, or people who suddenly retire and do something else to find more meaning once they have enough money to not worry about work.  I doubt you build a successful technology consulting firm that way.

On the other hand, my sense is that if you provide a real mission – not one of the fake mission statements that most companies have… but a real mission that matters, you will get levels of workplace engagement that exceed those workplaces where the mission is mercenary.

What happens when you have high levels of engagement?  You get more work out of people.  People row the boat harder.  They may work more hours, they may not (personally, I hope they don’t work more hours and burn out)… but you are highly likely to get more out of each hour from someone engaged in what she is doing than someone who is watching a clock.

So, if engagement is what helps you get the most from a given amount of human capital, and larger social goals and missions – beyond money drive engagement, it means that technology consulting companies must compete by persuing worthwhile social goals that drive engagement.  At scale, it also means these goals have to be near universal virtues (i.e. transcending religion, politics, etc.).

In the next post in the series, I will explore other businesses where this model is imporant, as well as types of businesses where frankly, this model does not work, due to dependence on attracting capital.

In Consulting, Human Capital Matters More than Financial Capital, Part 2