Is It Big Data, or Big Hype?

There is, of course, a kernel of truth to the hype. Much of the most interesting work in tech in general – and the startup world in particular – is occurring in what can vaguely be called the “Big Data” space. Amazon, Facebook, and Google, three of the so called “four horsemen of technology” (the fourth being Apple) drive their profitability via Big Data. As these companies lead, it isn’t just other startups following, but corporate IT departments are suddenly are looking at their long running “Business Intelligence” initiatives and wondering why they are not seeing the same kinds of return on investment. They are thinking…  if only we tweaked that “BI” initiative and somehow mix in some “Big Data”, maybe *we* could become the next Amazon.

Sadly, many such initiatives are doomed to failure. Many companies will task IT with coming up with a big data initiative, but won’t really involve the business at all. Many of these cases will result in IT going out and buying a product, installing it on desktops (or perhaps even tablets), and subsequently declaring victory. Of course, the value from this activity is dubious at best, typically resulting in lots of licenses acquired that, ultimately, sit on a shelf and seldom get used.

That might be bad, but there are worse things that can happen. There will be others that will go forth and try to build a comprehensive platform. Because IT often works in isolation, without a specific business problem to work on, the urge is often to try to build a solution that a theoretical business person can use to do analytics in a generic sense. There are, of course, serious problems with this approach:

• Tendency to spend years “perfecting the universal platform”.
• Building a platform that, in an attempt to be usable by a general business user, dumb things down and don’t deliver sufficient value.

Without a specific business problem to focus on, not only do you build more platform than you need, but you tend to build a platform that lacks the depth needed to solve the kind of problems you solve with modern analytical tools. Let’s face it – most business users are not experts in how to use Monte-Carlo simulation, neural networks, or other tools that are in the domain of the data scientist.

Want To Do Analytics? Start with a Business Problem!

It seems like rather trite advice, and I am loathe to say never very often, but I can say with certainty that you should never build out a Big Data or Analytics initiative without a specific business problem in mind. Sure, there are lot’s of people who would love to raid your corporate treasury in order to build out the “one platform to rule them all” – or even just sell you a bunch of shelfware. But that doesn’t actually do what you are there to do, which is get results, not buy software.

Our advice is at ThoughtWorks is, unambiguously, start small. Every good analytics problem has an underlying analytics question – something like “given what we know about a customer, how likely are they to leave for a competitor?” or “given a set of transactions, what is the likelihood of fraud?”. We then look at what data sources – some conventional, some far from conventional – are available to help solve the problem. We do discovery work in the data, and proceed to work up a hypothesis about how we can use said data to predict something about the customer, transaction, or other subject of interest. And then we test our hypothesis. And if the test works out, we move on and find a way to operationalize our finding for the benefit of the customer. Or if not, we seek the next hypothesis and try again.

The key to success in any agile process is the feedback loop. The problem with BI and even many incarnations of more modern analytics initiatives is the distinct lack of feedback that occurs when you build a platform before you solve a problem. The reason we call this Agile Analytics is because we use these feedback loops – and the learning associated with them – to guide our efforts.

Data Science versus Data Voodoo

Of course, a feedback loop doesn’t guarantee results. The work, the science, has to be solid as well. The tools of yesterday, doing things like building data cubes to slice and dice data, might be good at telling you what happened. They do little, however, to tell you meaning behind data, or to be useful for predictive use. For this, we bring in data scientists, often people with PhDs in mathematics, physics, or related fields, to develop predictive algorithms. The kind of people who developed things like modern spam filters that predict an answer to the question “is this email likely to be spam” using Bayesian classifiers.

The tools of the data scientist are vast indeed. Neural networks, machine learning, natural language processing, and more are among the techniques such people often use when developing analytics solutions. More importantly, they know when to use these tools, and they know when simpler tools will suffice – and when a pivot from one technique to another is needed if a hypothesis does not work out. It is the data scientist, not the tools themselves, that makes Agile Analytics possible. This is why we cringe when we see products advertised that claim to allow end users to bypass the data scientist and allow users of all proficiencies to, say, apply neural networks to data sets. We would feel the same way about do it yourself surgery kits!

Scaling Based on Results

The pundits are right about the potential around analytics and big data. But they frequently understate the risks of wasting money by engaging on long running programs that invest far too much prior to seeing results. ThoughtWorks took up Agile because we saw companies engage in massive waste by investing money into projects for years before results are realized. Analytics is no different. In fact, given the amount of hype around analytics, it is even more prudent you demand results before you scale up investment.

Our call to action is to start with discovery. We are happy to help with this, but even if we don’t, we believe starting these initiatives using small teams of less than six people. Start by establishing an analytics question, engage in a short discovery phase – typically around 3 weeks – to form and test your hypothesis, and based on your results, go from there. When Google got started, they certainly did not start with a seven figure budget and some enterprise software package promising to “index the internet”. Neither should you, as you engage in your analytics initiative.

Lately, as I have taken on additional roles at ThoughtWorks, the temptation to go “post technical” has put itself forward. Imagine not having to think anymore. Imagine being able to just focus on “strategy” and “people issues”, without all that hard technology stuff.

I could take that path, but I’d rather not…

I remain technical. My day-to-day job may involve things like Statements of Work and such, and I do not write code 100% of the time, but I am making a decision to at least remain somewhat involved in the technical communities in which I have interest.

Of course, my time is more limited, given I have taken on some management responsibility lately. So I can’t pursue everything. And I do have to admit that with my reduced time, I am likely never to be the most “technical person” in any given group of TWers. In fact, given that TWers are all generally really good at their jobs, it would be utter arrogance for me to assume I could keep up with them when I only act in a technical role for 40% of my time. So my bar isn’t TWers – they will most likely all be more technical than I am. But it is remaining competent enough to understand at least 40% of what they talk about, and more importantly, remain excited about technology.

So what am I excited about these days, here is my own, personal, “tech radar” (no fancy graphics required):

• Functional Languages and Big Data

Someday, the world will catch up to where the F#/Clojure people are now. I think the tie between FP and Big Data is strong, and given that is where much of the value will be created in the next 5 years with corporate IT spend, the use of FP will only increase.

• IaaS (note, I don’t say cloud… too overloaded)

I got the cloud bug this year, and I will admit, I am a terminal case. I saw an app where someone deployed a load balancer with an http request. And not an http request that converts into an email that goes to a tech that puts a physical one on a rack. Infrastructure as code is changing this business. Imagine the ability to specify, in human readable code, an organization’s entire server configuration. And realize that configuration by executing the DSL it is written in. That day is approaching, fast. Imagine the possibilities – from everything to devops to disaster recovery.

Why don’t I say cloud? The term is so overloaded that it is meaningless. Anything that runs on the internet is putting “cloud” in front of it’s name, which is why I have stopped using the term and do everything I can to use more descriptive terms.

• Lean Startup

Though not a technology, it is the first thing from a project management standpoint I have been truly excited about in years. At least since Agile, maybe more so. Why? It finally closes the loop, bringing in the entire scientific method into the business of software development. It seems so obvious now, it never ceases to surprise me that it took our industry 50 years to adopt it. Lean Startup, to me, is the application of the scientific method to how you run business. I recently did a webinar that nicely encapsulates how I feel about it in much more detail, but my general sense is that this builds on CD and gives organizations that embrace it a much more repeatable, sustainable path to successfully delivering business results with software.

The three things above are the tools I believe will create most of the value in corporate software over the next 5 years. And I fully intend to remain competent enough to discuss these topics with both technical and business audiences, even if I don’t write code every single day anymore. The trick, not being a full time software developer anymore, will be to:

• Make sure I don’t stop doing some technical work – I still intend to pair program with the team when I can, as well as keep up with a selection of OSS projects I am involved with.
• Continue to read new technical material – I am on planes a lot, should not be too hard. For example, I am learning Python now, despite that I don’t have a good reason to personally code in python.
• Be aware – ultra-aware – that I am not the most technical person in the room. My team-mates will likely know more than me, and I will defer to them on technical arguments more often than not, especially if they have decent data on why a given technical decision needs to be one way or another.

Will this work? We will see. I am only in the starting stages of this journey – will be interesting to see the degree to which my technical skills atrophy from not being a full time developer anymore!

Reason #1: Lack of an “adult/child” dynamic between sales and delivery that is common at many firms

In many companies, particularly consulting companies, salespeople call the shots, have most of the respect, and take home most of the rewards. Delivery – the “programmers”, are the people who are to be managed, paid as little as possible, and controlled by management. You see this commonly in pay structures, you see this in who is listened to when it is time to try a new idea (i.e. think agile before it was popular), and you see this in who makes up the management of the company. Bluntly, what you tend to see in most consulting companies is that former salespeople make up most of the management – which helps contribute to this effect.

At ThoughtWorks, in my experience, the relationship between demand generation and supply is much more balanced. It isn’t always perfect – sometimes we may go too far to the delivery end of the spectrum here, to the chagrin of our demand generation people – but it is much better than the status quo most places, where salespeople are considered “the adults in the room”, with “resources” (aka delivery) to be managed. It has a much more egalitarian feel – which feeds into a greater sense of career satisfaction and engagement. It is not a mistake that this greater sense of engagement, frankly, is a driver of our growth.

Reason #2: We have a purpose

In most companies, consulting or no, the sole purpose of profit is to enrich shareholders. At ThoughtWorks, we have a greater purpose. When we say we want to change the world and make it a better place, we mean it. As a firm, we invest our energy, our time, and significant resources towards projects that serve the cause of social justice.

Why does this matter? I don’t know about you, but the idea that the profits that I help create for this company make the world a better place is a lot more compelling to me than the alternative, which is usually helping some rich dude buy a slightly larger yacht. It makes me more engaged in my work than I would otherwise be. And I can’t speak for everyone at ThoughtWorks, but this is true for scores of others as well.

Ironically, it is this greater engagement that increases our financial returns. Companies that lack this purpose driven engagement often have to use extrinsic means (usually cash) in order to try to achieve the same level of commitment. Because our people are more engaged, they are more likely to go above and beyond for our clients. This, in turn, drives greater financial success, greater technical innovation, and greater ability to do social impact work – starting the cycle anew. While ThoughtWorks isn’t Consultopia, a concept I described in my book about consulting, it is likely the closest company among large technology consulting companies you will find.

Reason #3 – We are truly transnational

For me, one of the chief reasons I joined ThoughtWorks was that I wanted to work for a truly global company that was global for the right reasons. There are many companies that seek to open offices in India or other offshore locales in order to do rate arbitrage. They will say they are opening offices in places like India to access the talent pool, but what they usually mean is that they need a means to do work at lower rates.

One of the things that impresses me about ThoughtWorks is that, as a strategy, when we open an offshore office, we make it a priority to find work in that market, and as a goal, make it so that market can be self-sustaining. Two years ago, we opened our first Latin American office in Porto Allegre, Brazil. Within a year, we already had clients in the Brazilian market. Our efforts in China over the years have also yielded “in China, for China” type work.

There are several benefits to this. A primary benefit is that we gain better geographic diversity in our client base. This diversity allows us to better weather the natural ups and downs that will occur in any given region. As places like Brazil, India, and China rise, we already have relationships with those respective business communities. This does not happen if you are primarily in the rate arbitrage business, as companies that do that tend to tie offshore revenue to first world economies – spending most of their energy doubling down on first world business relationships, rather than developing world relationships.

Why is this good for you, as a potential ThoughtWorker? First off, you get a chance to get involved in these emerging markets. Second, you have a better shot at getting exposed to these cultures – all of which add to your value as a consultant should you decide to learn how to work with such a diverse set of people. Third, you work for an organization that is sufficiently diversified that the potential rise of Brazil, India, and China will only create more opportunities for you, rather than fewer.

Reason #4 – We won’t do evil

As a company, we have strong values. Every client we take on, we vet to make sure we are comfortable with what they do as a business. This isn’t always easy, and there are definitely shades of gray. But chances are, if a company derives most of it’s revenue from activities related to war, uses it’s profits to fuel hate against marginalized minorities, such as blacks, women, or the LGBT community, or otherwise does not have aligned values, we will not work for them.

This does not come without cost. There have been times where our pipeline was light, and work was offered that would help the company through a slow period. And frequently, there are very vibrant conversations about exactly where to draw the line when something is a “shade of gray”. I am reasonably certain we do not always get it right. But compared to most companies, which, being polite, operate on the principle of ATM (Anything For Money) – ThoughtWorks is head and shoulders above the rest.

Reason #5 – We often work on the hardest, most difficult problems

When we engage, the stakes are usually quite high for the company engaging us. This provides a sense of meaning and purpose to the work that is compelling, frequently very technically interesting, and almost always very interesting from a social point of view. After approaching three years at ThoughtWorks, it has become rather obvious to me that many problems that are presumably software problems really have a corporate strategy or political problem at their core. This kind of work, very common here, helps you as a ThoughtWorker exercise skills that a pure software shop can’t offer – things like building coalitions, selling crazy ideas, and creating delivery climates conducive to innovation. ThoughtWorks is not just a great place to learn technology, agile, lean, or continuous delivery – it is also a great place to learn how to navigate the ropes that exist in large organizations so you can actually get things done.

It isn’t for Everyone

Look, ThoughtWorks isn’t for everyone. Most roles require extensive travel. And we have a pretty hard-core vetting process for new hires. But I believe that the opportunity to do amazing work – work that is literally “make or break” type work for many of our clients – in a manner that truly makes the world a better place – is worth it. It is a place where you will have deep respect as a technical person. It is a place that bears the costs of doing the right thing. It is a place that you will be proud to work for. If you would like to join us, please let me know, either directly (aerickson – at – thoughtworks.com), or by going to our new site for potential candidates at join.thoughtworks.com.

So let’s turn back the clock through the mists of time in the years leading to the Agile Manifesto in 2001. Back when this movement started, many of the “reasons for Agile” just seemed very intuitive. People over process? Sure. Responding to change versus following a predefined plan? Common sense. On the surface, few people would disagree with the assertions made in the Agile Manifesto. But is that enough? Can you following the Manifesto—do it all by the book—and be guaranteed to create software that delivers economic benefit? No. By itself, Agile doesn’t lead to business value!

How Can By-the-Book-Agile Fail?

Let’s be clear. With its increased collaboration with the customer, more frequent releases, and increased engineering and testing discipline, Agile makes delivering value more likely. It’s certainly a vast improvement over multi-year “too big to fail” Waterfall software projects. But even if you do everything right, even if you have the best practitioners in the world building your product, you can still create a product that fails to make money.

Agile—and here we mean the kind that includes the right engineering practices, such as test-driven development (TDD), pairing, SOLID principles, automated acceptance testing, and continuous integration—will ensure that you create a product that works…in the lab, at least. However, despite your best efforts, you might build a very functional eCommerce site that tries to sell something that nobody really wants. Or you might build an internal application that can’t go beyond the lab because the operations people can’t support it in production. These issues—and many other things outside direct control of the team practicing Agile—can thwart even your best efforts.

Article continues here

One problem though. Your rates are too high. Can you help us train this group over here to be like you? They work for 1/3 your cost, and don’t ask us lots of hard questions all the time.

Sigh.

Of all the ways to sell services of higher end technology consultants – if I had to choose a worst way – I think I would end up with hourly rates. It is the same measurement vendor management departments in most large companies use to “price” the services of said consultants.  Unfortunately, it is a number that, in isolation, tells you very little of importance, for two main reasons:

• Measurement of effort at the wrong level of precision
• Measurement of cost irrespective of value delivered

Measurement of effort at the wrong level of precision

How many people are doing estimates on a per hour basis? There may be a few, but I highly doubt there are people other than GANTT chart driven control freaks looking to normalize that kind of thing. The reality is that, as a developer, I have a mix of useless hours, and a few really good ones where the vast majority of the value get’s delivered. The whole hours thing is a vestige from the temporary employment industry where programming was considered a fancier form of typing. We need to recognize it for the vestigial tail it is.

What should replace it? I am a big fan of weekly or monthly rates when working under a time and materials contract. I don’t mind the idea of “per person-sprint” either – which fits nearly into the idea that we would rarely fund only a percentage of a given sprint or iteration. I am open to other models that match funding with the unit of work, but do so in a way that you can know in advance (i.e. I would reject per story on the basis of you can only estimate effort – and it brings on too many potential fights or negotiations about how big a story ended up being).

Measurement of cost irrespective of value delivered

The precision issue, however, isn’t the biggest issue. The biggest issue I have with hourly rates is that they are the denominator in a much more important number, that being ratio of value delivered to cost. This issue is an old one, well covered in Joel Spolsky’s post on the subject. The specific problem – however – for higher end consulting firms, is that rates tend to be higher. Without considering value delivered in part of the equation, rate conversations tend to always be self-defeating, as it highlights the cost, without talking about the value.

The problem, of course, is that value delivered is often a squishy thing to measure, while you can read the cost per hour right off the invoice. To me, this is the biggest reason why Continuous Delivery is important, particularly the kind that has a feedback loop that measures financial performance – it allows us to actually measure the numerator in the value delivered equation. In the best of all worlds, we move to a funding model that funds the Minimum Viable Product up front, but then moves to a Evidence Based Funding model (aka “Continuous Funding”) based on value delivered via Continuous Delivery.

Something Has To Be Done

The funding model is broken. It has been for a long time, leading to sub-optimal results. Competition by rate irrespective of value delivered is a race to the bottom, leading to all sorts of bad outcomes for companies that get involved. Sadly, too many do. Continuous Delivery is a start, perhaps if we can get to Evidence Based Funding – we can actually sustain continuous delivery.

It’s amazing how things change in ten years. Once considered a methodology preferred by software developers because it “helped them avoid having to do status reports,” Agile has now gone beyond its original remit as a software development method. The word Agile, in many places, has become nearly synonymous with the word Good. As flattering this must be to the original founders, it probably means it would be wise to think, very candidly, whether Agile—as in Agile Software Development, or a broader Agile Enterprise—is really something that your company can achieve.

Read the rest of this article at InformIT here.

As it seems to have turned out, we are now in the Revenge of The Nerds economy. Despite 9%+ unemployment in the general economy, overall unemployment for software engineers is a tad below 5%. But this data, which is compelling enough, does not tell the entire story.

The bigger story is around what kinds of things are seeing investment. There is talk of another bubble right now in technical startups. If you are a technology based startup that has reached “Ramen Profitability”, you will almost certainly attract capital. Things are now even getting to the point where we are seeing companies that lack profits going to the IPO market (think Pandora and Groupon) – something that has been out of vogue since at least 2001. One could make the case that we will soon cross the line where it is easier to get a startup funded than it is to get a jumbo mortgage.

If you work in tech, there is a good chance you are feeling this, at least if you are currently employed and working for an employer that has some level of visibility. If you work for Google, Facebook, or some other high tech company, you likely do not have to work that hard to get a new job offer. Notwithstanding the cruel irony that people who are unemployed are often being systemically discriminated against, the job market for really good, currently employed software developers is as robust has it has been in 10 years.

So if the nerds are all right, mostly, what about the jocks? What occupations did they end up in? While some of them may have made it to the NFL as professional football players, most others end up in spilling into the general job market. While the stereotype is that jocks end up in menial jobs, construction, or manufacturing; research being hard to find, my own experience of knowing several such folk points to careers tending to be sales, low-end finance (think mortgages), real estate, and personal training. Given, a sample size of a couple dozen doesn’t really prove anything. However, if one did extrapolate, one could come to a conjecture that while “jocks” for lack of a better word, do not do worse then average now, the nerd/jock investment and employment dynamic has changed.

Why the change? Now that simply taking big leveraged risks with a big pile of money isn’t in fashion (i.e., as it was before the global financial crisis), you need advantages from superior intellectual capital in order to sustain a higher than mean return on investment. Why not in fashion? From 2003-2008, it was accepted that financial engineering was the way to riches. If you only structured your collateralized debt obligation in the correct way, you could invest at 40x leverage and still retain a AAA rating on your debt. Financial engineering gave you higher profits under such a regime than traditional “engineering engineering” could provide. So money flowed into CDS structures and away from the nerdy parts of the economy that invents things.

So now we find ourselves in an economy where capital flocks to things like Pandora, Groupon, Facebook, Linkedin, and other things that, at their core, have interesting algorithms inside them. And we have a situation where if you are a developer capable of writing an interesting and valuable algorithm – or helping a company scale out a system that leverages one of these inventions, you are in high demand.

Good thing for the nerds. What this means for everyone else remains to be seen.

Without proper engineering practices, you will not see the benefits of agile software development.  It comes down to basic engineering principles.  A highly coupled system – be it software, mechanical, or otherwise, provides more vectors over which change in one part of the system can ripple into other parts of the system.  This is desirable to a degree – you need your transmission to couple to the engine in order for the engine to make a car move.  But the more coupling beyond the minimum you need in order to make the system work, the more the overall system destabilizes.  If the braking system suddenly activated because a Justin Bieber CD was inserted into the car stereo, you would probably see that as a pretty bad defect.  And not just because of the horrible “music” coming out of the speakers.

So what are the specific engineering practices?  Some are lower level coding practices, others are higher level architectural concerns.  Most rely on automation in some respect to guard against getting too complacent.  While I generally loathe to use the term “best practices” due to the fear that someone might take these practices and try to apply them in something of a cargo-cult manner, these are some general practices that seem to across a broad section of the software development world:

Test Driven Development and SOLID

While detractors remain, it has ceased to be controversial to suggest that the practices that emerged out of the Extreme Programming movement of the early 2000s are helpful.  Test driven development as a design technique selects for creating decoupled classes.  It is certainly possible to use TDD to drive yourself to a highly-coupled mess, given enough work and abuse of mocking frameworks.  However, anyone with any sensitivity to pain will quickly realize that having dozens of dependencies in a “God” class makes you slow, makes you work harder to add new functionality, and generally makes your tests brittle and worthless.

To move away from this pain, you write smaller, testable classes that have fewer dependencies.  By reducing dependencies, you reduce coupling.  When you reduce coupling, you create more stable systems that are more amenable to change – notwithstanding the other benefits you get from good test coverage.  Even if you only used TDD for design benefits- and never used the tests after initially writing them, you get better, less coupled designs, which leads to greater velocity when you need to make changes.  TDD doesn’t just help you in the future.  It helps you move faster now.

Indeed, TDD is just one step on the way to keeping your code clean.  Robert Martin treats the subject in much more depth in his book, Clean Code.  Indeed, he calls all of us out to be professionals, making sure that we keep our standards and don’t give into the temptation to simply write a bunch of code that meets external acceptance criteria, but does so at the cost of internal quality.  While you can, in theory, slap some code together that meets surface criteria, it is false economy to assume that bad code today will have a positive effect on velocity beyond the current iteration.

Of course, having good test coverage… particularly good automated integration, functional, performance, and acceptance tests, has a wonderful side effect of forming a robust means of regression testing your system on a constant basis.  While it has been years since I have worked on systems that lacked decent coverage, from time to time I consult for companies that want to “move to agile”.  Almost invariably, when I do this, I find situations where a captive IT department is proposing that an entire team spend 6 months to introduce 6 simple features into a system.  I see organizations that have QA departments that have to spend another 6 months manually regression testing.  TDD is a good start, but these other types of automated testing are needed as well to keep the velocity improvements going – both during and after the initial build.

Simple, but not too simple, application architecture (just enough to do the job)

While SOLID and TDD (or BDD and some of the ongoing improvements) are important, it is also important to emphasize that simplicity specifically as a virtue.  That is not to say that SOLID and TDD can’t lead to simplicity – they certainly can, especially in the hands of an experienced practitioner of the tool.  But without a conscious effort to keep things simple (aka apply the KISS principle – keep it simple and stupid), regardless of development technique, excess complexity creeps in.

There are natural reasons for this.  One of which is the wanna-be architect effect.  Many organizations have a career path where, to advance, a developer needs to advance to the title of architect – often a role where, at least it is perceived, that you get to select design patterns and ESB buses without having to get your hands dirty writing code.  There are developers who believe that, in order to be seen as an architect, you need to use as many GoF patterns as possible, ideally all in the same project.  It is projects like these where you eventually see the “factory factories” that Joel Spolsky lampooned in his seminal Architecture Astronaut essay.  Long story short, don’t let an aspiring architecture astronaut introduce more layers than you need!

It doesn’t just need to be a wanna-be architecture astronaut that creates a Rube Goldbergesque nightmare.  Sometimes, unchecked assumptions about non-functional requirements can lead a team to creating a more complex solution than actually needed.  It could be anything from “Sarbanes Oxley Auditors Gone Wild” (now there is a blog post of it’s own!) requiring using an aggressive interpretation of the law to require layers you don’t really need.  It could be being asked for 5 9s of reliability when you only really need 3. These kinds of excesses show up all the time in enterprise software development, especially when they come from non-technical sources.

The point is this – enterprise software frequently introduces non-functional requirements in something of a cargo-cult manner “just to be safe”, and as a result, multiplies the cost of software delivery by 10.  If you have a layer being introduced as the result of a non-functional requirement, consider challenging it to make sure it is really a requirement.  Sometimes it will be, but you would be surprised how often it isn’t.

Automated Builds, Continuous Integration

If creating a developer setup requires 300 pages of documentation, manual setup, and other wizardry to get right, you are likely to move much slower.  Even if you have unit tests, automated regression tests, and other practices, lacking an automated way to build the app frequently results in “Works on My Machine” syndrome.  Once you have a lot of setup variation, which is what you get when setup is manual, defect resolution goes from straightforward process to something like this:

1. Defect Logged by QA
2. Developer has to manually re-create defect, spends 2 hours trying, unable to do so
3. Developer closes defect as “unable to reproduce”
4. QA calls developer over, reproduces
5. Argument ensues about QA not being able to setup the environment correctly
6. Developer complaining “works on my machine”
7. 2 hour meeting to resolve dispute
8. Developer has to end up diagnosing the configuration issue
9. Developer realizes that DEVWKSTATION42 is not equivalent to LOCALHOST for everyone in the company
10. Developer walks away in shame, one day later

Indeed, having builds be automated, regular, and integrated continuously can help avoid wasting a day or five every time a defect is logged.  It should not be controversial to say that this practice increases velocity.

Design of Software Matters

Design isn’t dead.  Good software design can help lead to good velocity.  Getting the balance wrong – too simple, too complex, cripples velocity.  Technical practices matter.  Future articles in this velocity series will focus on some of the more people related ways to increase velocity, and they are certainly important.  But with backward engineering practices, none of the things you do on the people front will really work.

“Custom software costs too much!”

There are stories, anecdotes, studies, and all sorts of experience about how software development schedules that go over time, well over budget, and fail to meet expectations.  Far more seldom are stories that talk about how software was delivered in less time than was estimated.  It should be no surprise that, despite the fact that corporations are sitting on more cash than they ever have in corporate history, companies are still skittish about embarking on new custom development projects.

This, of course, is not new.  This has been a theme in software development even far before Fred Brooks wrote The Mythical Man Month.  What is surprising to me – is how this condition survives even despite things like agile and lean coming to the fore.  Yes, its true, even in purported agile and lean projects, there is a great variability in productivity that, when combined with a tendency to budget and plan based on best case scenarios in corporate IT, result in far too many late and over budget projects.  Agile and lean help – indeed, are the right direction (this is one of the reasons I work for ThoughtWorks!). However, they are not a silver bullet.  The reality is that, despite everything we try to do to mitigate risks with agile and lean, project sponsors are rightly concerned with the level of risk they see as inherent in software projects.  As a result, they are going to use levers available to them to manage these risks.

Levers Used to Manage Risk

So what levers do investors have in order to manage risk?  The ones that most use tend to be:

• Manage using process controls (scope management, quality gates, governance, PMBOK, ITIL)
• Manage down the invested capital at risk

What happens when you pull these levers?  Well, sadly, this is what you tend to get:

• Waterfall software development, often fixed bid, with strict “de jure” scope control with the real “de facto” scope additions added through a back-channel.
• Attempt to get lowest cost per developer hour.

While this does not occur in more enlightened organizations that likely understand the fact that not all “developer hours” are the same, the sad fact is that most organizations lack such enlightenment.  It is easy to imagine why this occurs.  If you are a CIO managing a 2000 person IT department, part of your bias is going to be that you, being a smart CIO and all, have such a large organization for a reason.  At this level of corporate politics, not only is your prestige buried in the size of your department, but the idea that you have 80% more people than you might need is, to put it simply, heretical.  Given that such heretics that “rock the boat” tend to have short careers in high level corporate IT, it is not surprising that the such “resource oriented” ways of thinking predominate the conversation at the highest levels of IT.

Why These Levers Fail

Not all failure is the same.  And not all failing is bad.  The problem is when we fail to learn from failure.  And the way these levers fail, then tend to encourage the kind of failure you don’t learn from.  The kind of failure that begets more failure rather than begets learning from mistakes.

How the Hourly Rate Lever Fails

Let us take on failure that results when you try to succeed through usage of the cost reduction lever.  A common way this gets sold is when there is a vendor management group – especially one where the leading VP is bonused strictly on lowering rates – that wields a lot of power in an organization.  The presence of companies selling developers chiefly on the basis of rate attests to the fact that there is a market for this.  You do not have to look hard for this – much of the market for offshore development caters to this group.  Not to be outdone, there are many first world (US in particular) based companies that try to play the low end of the market.

However, another source of “search for the lowest rate” surprisingly comes from initial (and often feeble) attempts to try to get estimating right as a result of a failed waterfall project.  If the last project failed, the chances are, the next round of estimating will be much more conservative.  The conservative estimate, often higher than initial budget expectations, causes the sponsor to try to drive down the unit cost per hour.  Remember, in such a situation, the hour is perceived as the fixed quantity.  This attempt to fit a large number of hours into a fixed budget also is a large source of project sponsors trying to pull the hourly rate lever.

Predictably, these kinds of moves fail.  When you go lower on the rate scale, you tend to get less qualified software developers.  While there is a lot of corporate hubris tied into the fiction that there is some magical source of low cost developers that are otherwise brilliant and able to achieve results above their pay grade, seldom do such moves tend to work out.  Low cost shops tend to have lots of turnover because of low wages – good developers are not stupid, and will not tolerate low wages for long.  Beyond this, of course, low cost shops tend to underestimate work as well, because they often compete with other low cost shops.  It is a race to the bottom, ironically, that causes even more project failure.

The result of all this is, of course, chasing low-cost development tends to cause a self-reinforcing failure cycle.  Project fails, next project has longer estimates, sponsor does not want to spend more money on failure, leading to the next push to drive rates even lower.  This continues until the organization loses any faith in any ability to write software at all.  The only thing that stops this is, literally, company bankruptcy, acquisition, or a major change in management at the top of IT (if not the CEO) so that the negative feedback cycle can be broken.

How the “Strong Process” Lever Fails

One step up from project sponsors who will hit the cost lever like a mouse hitting a lever in a skinner box are the project sponsors who put their full faith in processes, such as those proscribed by the PMBOK or recommended by ITIL to solve their cost problems.  The thought is that – if only we could get a better plan, and stick to the plan, we could get better at predicting the cost of software.

The chief thing that such frameworks try to do is to do strong change management, contracted service levels, and detailed planning in order to achieve the end of predictable software development.  This approach, of course, leads to several problems that are well understood by many people in the Agile and Lean communities but certainly bear repeating:

• Software development itself, particularly when done with business analysts, frequently leads to discoveries.  As Mary Poppendeick once pointed out, the most valuable feedback for changes to software comes at the latter end of a project, when the group analyzing and developing the software best understands the domain.  Traditional processes tend to consider these late changes the most costly, seeking to minimize the highest value work.
• Velocity is deeply variable based on factors that management usually does not want to talk about.  A good software developer can be 10 times more effective than an average software developer.  Other environmental factors that have little to do with the presence of a prescriptive process can cause additional wild swings.  A future article in this series will go into more detail about how potential swings in velocity are an order of magnitude more significant than the swings in “cost per hour”.

Proscriptive plans that try to predict cost and schedule without understanding the individuals, service climate, DevOps maturity, and multitude of other variables that can affect the speed of software development will surely fail.  The simple fact that corporate budgeting processes select for optimistic thinking most of the time leads to assuming that simple program that consists of 4 main screens will take 8 weeks rather than 8 quarters.

Why is this a cycle of failure?  Well – what happens is that you bring PMBOK, ITIL, or some other “capital-P” Process in, and it fails.  Because such processes tend to be more accepted by the corporate finance community, the reaction to a failed project tends to be “we didn’t do enough process control”, rather than “we did too much”.  The next round of projects tend to have more detailed estimating and stronger change control.  There are stories that I personally know of requirements efforts that are measured by years, rather than weeks.

What Levers Remain?

There are levers that go far too much underused.  In my experience, they are as follows:

Manage Project Duration Down

In general, prefer smaller projects to larger ones.  Predictability works better when the team is smaller and the timeframe is short.  I can almost decently predict what one team of 8 people will complete in an iteration if I know all the people, skills, and modes of working on that team.  I, personally, am a believer in doing lots of small things rather than one big thing – as it allows for a faster feedback cycle and overall a less risky investment.  However, there are many projects – some of the most interesting ones – that will never be small budget, short duration projects.  This leads to the lever I use when I can’t manage duration down:

Manage Velocity Up

I have seen a team of 6 people deliver in 7 months in one company the same amount of work I have seen 150 people deliver in 2 years in other companies.  And I am not alone, I have talked with others in this business, and nearly everyone I talk to has either seen, or worked on one of these very effective software development teams.

Velocity matters.  It is hard to measure productivity the way you measure a person’s shoe size, but you can sure manage relative productivity over a long period of time.  The perception that software costs too much is, in my opinion, one of the factors holding back the global economy from expanding.  In any large company, any decent developer,  would be shocked at the waste involved in many business processes – waste that could be eliminated by better software implementation.  There are companies that will not change commission plans, product offerings, or even product pricing to be more profitable, chiefly because of software constraints.  If you think software does not have a serious effect on both the opportunities or limitations of a company, you literally must not be paying very much attention.

What’s Next?

In this series of upcoming posts, I am going to explain various factors that affect velocity, and how you can optimize them in a manner that does not sacrifice quality.  Factors that, when understood and properly managed, can mean the difference between a flexible company that can respond to market conditions, and a company whose technology that does not allow it to run so much as a second shift when demand ramps up.

Stay tuned…

Technology Consulting is 10% Technology, 90% Consulting

Being a great technologist is around 10% of the skillset required for being a good technology consultant.  I used to think it was 50%, but my understanding has now vastly changed towards the direction of the so called “soft skills” being more important.  In most companies, there are thousands of opportunities to make things better using technology in some way, shape, or form.  The trick to opening those opportunities is overcoming the massive “wall of cynicism” towards these kinds of investments.  Discovering the opportunities, overcoming the wall of cynicism, getting the human stakeholders on board (not just upper management!), and then actually putting this all together to get a project funded and delivered seems to be 90% of the challenge.

Technology Consulting is a Subset Of, not Different Than, Management Consulting

You can do a project without having it have deep implications for the overall business.  But I doubt that is the kind of project I would ever want to work on.  Most technology investments are, in fact, capital investments in the business.  It is very common for technology to end up both constraining and enabling corporate strategy. Even minor implementation details can have significant effects on strategic choices that a company will be able to make down the road. Everything from ability to do effective mergers, price changes, new product launches, and other strategic initiatives deeply depend on business technology.  For good or ill, technology decisions drive business strategy – and until that condition changes, it is my conjecture that effective technology consulting is really a part of the overall management consulting picture.

ThoughtWorks, having recognized this, is formally rolling out our Consulting Practice which, among other things, seeks to formally do what we have been doing informally since our inception, which is to advise companies not just on how to implement technology, but what technology to implement.

Clients are Never Perfect

Expecting to come to TW where you will work on perfectly “aligned” clients that always espouse our values is, to be blunt, a poor expectation. Most people call the doctor when they are sick, not when they are well. Sometimes – no – all the time – organizational transformation is HARD, and you will have to, excuse my French, slog through some shit in order to get to the holy grail of your client that needs help becoming a perfectly functioning agile client that perfectly practices continuous delivery.

Never Underestimate The Importance of Soft Skills

Good consulting involves:

• Controlling your temper and not being “shocked” when you see things like bad code and retrograde practices. When you go on site, expect anything and everything.
• Understanding and addressing the skepticism of organizations for which you are the 4th, 5th, or 10th person who has been put there to try to fix things.
• Learning how to build credibility so you can spend it later when you need to.
• Understanding the limits of your own capabilities, so you can know when to call for help (aka you do not have all the answers).
• Learning how to understand a domain quickly and credibly, so you can talk in the language of the client
• … and a million other little things that have more to do with relations between humans than they have to do with technology …

It has been quite an experience, personally, just finding out how much I didn’t know, learning how to apply these principles in a large programme of work.  That these things are important is rather intuitivley obvious, but at scale, it becomes a list of things you have to remind yourself about every day.  When the stakes go up, and the number of people increase, the importance of these basic fundamentals really starts to outweigh almost everything else!

The Bottom Line

If you want to actually deliver a great technology solution, getting the technology right is just the table stakes.  Getting the people thing right – the consulting – is 90% of the actual work.  It is thrilling, engrossing work, but it certainly isn’t just about software!