Velocity

IF one could equate faster typing with velocity, engineering practices perhaps would not matter in the world of software development productivity. Thankfully, there are reasons that most organizations do not use Words Per Minute in our evaluation process when hiring new software developers. Slamming out low quality code and claiming progress, be it story points, or merely finished tasks on a GANTT chart, is a fast track to creating boat anchors that hold back companies rather than pushing them forward.

Without proper engineering practices, you will not see the benefits of agile software development. It comes down to basic engineering principles. A highly coupled system – be it software, mechanical, or otherwise, provides more vectors over which change in one part of the system can ripple into other parts of the system. This is desirable to a degree – you need your transmission to couple to the engine in order for the engine to make a car move. But the more coupling beyond the minimum you need in order to make the system work, the more the overall system destabilizes. If the braking system suddenly activated because a Justin Bieber CD was inserted into the car stereo, you would probably see that as a pretty bad defect. And not just because of the horrible “music” coming out of the speakers.

So what are the specific engineering practices? Some are lower level coding practices, others are higher level architectural concerns. Most rely on automation in some respect to guard against getting too complacent. While I generally loathe to use the term “best practices” due to the fear that someone might take these practices and try to apply them in something of a cargo-cult manner, these are some general practices that seem to across a broad section of the software development world:

Test Driven Development and SOLID

While detractors remain, it has ceased to be controversial to suggest that the practices that emerged out of the Extreme Programming movement of the early 2000s are helpful. Test driven development as a design technique selects for creating decoupled classes. It is certainly possible to use TDD to drive yourself to a highly-coupled mess, given enough work and abuse of mocking frameworks. However, anyone with any sensitivity to pain will quickly realize that having dozens of dependencies in a “God” class makes you slow, makes you work harder to add new functionality, and generally makes your tests brittle and worthless.

To move away from this pain, you write smaller, testable classes that have fewer dependencies. By reducing dependencies, you reduce coupling. When you reduce coupling, you create more stable systems that are more amenable to change – notwithstanding the other benefits you get from good test coverage. Even if you only used TDD for design benefits- and never used the tests after initially writing them, you get better, less coupled designs, which leads to greater velocity when you need to make changes. TDD doesn’t just help you in the future. It helps you move faster now.

Indeed, TDD is just one step on the way to keeping your code clean. Robert Martin treats the subject in much more depth in his book, Clean Code. Indeed, he calls all of us out to be professionals, making sure that we keep our standards and don’t give into the temptation to simply write a bunch of code that meets external acceptance criteria, but does so at the cost of internal quality. While you can, in theory, slap some code together that meets surface criteria, it is false economy to assume that bad code today will have a positive effect on velocity beyond the current iteration.

Of course, having good test coverage… particularly good automated integration, functional, performance, and acceptance tests, has a wonderful side effect of forming a robust means of regression testing your system on a constant basis. While it has been years since I have worked on systems that lacked decent coverage, from time to time I consult for companies that want to “move to agile”. Almost invariably, when I do this, I find situations where a captive IT department is proposing that an entire team spend 6 months to introduce 6 simple features into a system. I see organizations that have QA departments that have to spend another 6 months manually regression testing. TDD is a good start, but these other types of automated testing are needed as well to keep the velocity improvements going – both during and after the initial build.

Simple, but not too simple, application architecture (just enough to do the job)

While SOLID and TDD (or BDD and some of the ongoing improvements) are important, it is also important to emphasize that simplicity specifically as a virtue. That is not to say that SOLID and TDD can’t lead to simplicity – they certainly can, especially in the hands of an experienced practitioner of the tool. But without a conscious effort to keep things simple (aka apply the KISS principle – keep it simple and stupid), regardless of development technique, excess complexity creeps in.

There are natural reasons for this. One of which is the wanna-be architect effect. Many organizations have a career path where, to advance, a developer needs to advance to the title of architect – often a role where, at least it is perceived, that you get to select design patterns and ESB buses without having to get your hands dirty writing code. There are developers who believe that, in order to be seen as an architect, you need to use as many GoF patterns as possible, ideally all in the same project. It is projects like these where you eventually see the “factory factories” that Joel Spolsky lampooned in his seminal Architecture Astronaut essay. Long story short, don’t let an aspiring architecture astronaut introduce more layers than you need!

It doesn’t just need to be a wanna-be architecture astronaut that creates a Rube Goldbergesque nightmare. Sometimes, unchecked assumptions about non-functional requirements can lead a team to creating a more complex solution than actually needed. It could be anything from “Sarbanes Oxley Auditors Gone Wild” (now there is a blog post of it’s own!) requiring using an aggressive interpretation of the law to require layers you don’t really need. It could be being asked for 5 9s of reliability when you only really need 3. These kinds of excesses show up all the time in enterprise software development, especially when they come from non-technical sources.

The point is this – enterprise software frequently introduces non-functional requirements in something of a cargo-cult manner “just to be safe”, and as a result, multiplies the cost of software delivery by 10. If you have a layer being introduced as the result of a non-functional requirement, consider challenging it to make sure it is really a requirement. Sometimes it will be, but you would be surprised how often it isn’t.

Automated Builds, Continuous Integration

If creating a developer setup requires 300 pages of documentation, manual setup, and other wizardry to get right, you are likely to move much slower. Even if you have unit tests, automated regression tests, and other practices, lacking an automated way to build the app frequently results in “Works on My Machine” syndrome. Once you have a lot of setup variation, which is what you get when setup is manual, defect resolution goes from straightforward process to something like this:

Defect Logged by QA
Developer has to manually re-create defect, spends 2 hours trying, unable to do so
Developer closes defect as “unable to reproduce”
QA calls developer over, reproduces
Argument ensues about QA not being able to setup the environment correctly
Developer complaining “works on my machine”
2 hour meeting to resolve dispute
Developer has to end up diagnosing the configuration issue
Developer realizes that DEVWKSTATION42 is not equivalent to LOCALHOST for everyone in the company
Developer walks away in shame, one day later

Indeed, having builds be automated, regular, and integrated continuously can help avoid wasting a day or five every time a defect is logged. It should not be controversial to say that this practice increases velocity.

Design of Software Matters

Design isn’t dead. Good software design can help lead to good velocity. Getting the balance wrong – too simple, too complex, cripples velocity. Technical practices matter. Future articles in this velocity series will focus on some of the more people related ways to increase velocity, and they are certainly important. But with backward engineering practices, none of the things you do on the people front will really work.

After nearly 20 years writing custom software, mostly for corporations in IT departments, there is nearly a uniform meme I encounter among those who sponsor projects that involve custom software development:

“Custom software costs too much!”

There are stories, anecdotes, studies, and all sorts of experience about how software development schedules that go over time, well over budget, and fail to meet expectations. Far more seldom are stories that talk about how software was delivered in less time than was estimated. It should be no surprise that, despite the fact that corporations are sitting on more cash than they ever have in corporate history, companies are still skittish about embarking on new custom development projects.

This, of course, is not new. This has been a theme in software development even far before Fred Brooks wrote The Mythical Man Month. What is surprising to me – is how this condition survives even despite things like agile and lean coming to the fore. Yes, its true, even in purported agile and lean projects, there is a great variability in productivity that, when combined with a tendency to budget and plan based on best case scenarios in corporate IT, result in far too many late and over budget projects. Agile and lean help – indeed, are the right direction (this is one of the reasons I work for ThoughtWorks!). However, they are not a silver bullet. The reality is that, despite everything we try to do to mitigate risks with agile and lean, project sponsors are rightly concerned with the level of risk they see as inherent in software projects. As a result, they are going to use levers available to them to manage these risks.

Levers Used to Manage Risk

So what levers do investors have in order to manage risk? The ones that most use tend to be:

Manage using process controls (scope management, quality gates, governance, PMBOK, ITIL)
Manage down the invested capital at risk

What happens when you pull these levers? Well, sadly, this is what you tend to get:

Waterfall software development, often fixed bid, with strict “de jure” scope control with the real “de facto” scope additions added through a back-channel.
Attempt to get lowest cost per developer hour.

While this does not occur in more enlightened organizations that likely understand the fact that not all “developer hours” are the same, the sad fact is that most organizations lack such enlightenment. It is easy to imagine why this occurs. If you are a CIO managing a 2000 person IT department, part of your bias is going to be that you, being a smart CIO and all, have such a large organization for a reason. At this level of corporate politics, not only is your prestige buried in the size of your department, but the idea that you have 80% more people than you might need is, to put it simply, heretical. Given that such heretics that “rock the boat” tend to have short careers in high level corporate IT, it is not surprising that the such “resource oriented” ways of thinking predominate the conversation at the highest levels of IT.

Why These Levers Fail

Not all failure is the same. And not all failing is bad. The problem is when we fail to learn from failure. And the way these levers fail, then tend to encourage the kind of failure you don’t learn from. The kind of failure that begets more failure rather than begets learning from mistakes.

How the Hourly Rate Lever Fails

Let us take on failure that results when you try to succeed through usage of the cost reduction lever. A common way this gets sold is when there is a vendor management group – especially one where the leading VP is bonused strictly on lowering rates – that wields a lot of power in an organization. The presence of companies selling developers chiefly on the basis of rate attests to the fact that there is a market for this. You do not have to look hard for this – much of the market for offshore development caters to this group. Not to be outdone, there are many first world (US in particular) based companies that try to play the low end of the market.

However, another source of “search for the lowest rate” surprisingly comes from initial (and often feeble) attempts to try to get estimating right as a result of a failed waterfall project. If the last project failed, the chances are, the next round of estimating will be much more conservative. The conservative estimate, often higher than initial budget expectations, causes the sponsor to try to drive down the unit cost per hour. Remember, in such a situation, the hour is perceived as the fixed quantity. This attempt to fit a large number of hours into a fixed budget also is a large source of project sponsors trying to pull the hourly rate lever.

Predictably, these kinds of moves fail. When you go lower on the rate scale, you tend to get less qualified software developers. While there is a lot of corporate hubris tied into the fiction that there is some magical source of low cost developers that are otherwise brilliant and able to achieve results above their pay grade, seldom do such moves tend to work out. Low cost shops tend to have lots of turnover because of low wages – good developers are not stupid, and will not tolerate low wages for long. Beyond this, of course, low cost shops tend to underestimate work as well, because they often compete with other low cost shops. It is a race to the bottom, ironically, that causes even more project failure.

The result of all this is, of course, chasing low-cost development tends to cause a self-reinforcing failure cycle. Project fails, next project has longer estimates, sponsor does not want to spend more money on failure, leading to the next push to drive rates even lower. This continues until the organization loses any faith in any ability to write software at all. The only thing that stops this is, literally, company bankruptcy, acquisition, or a major change in management at the top of IT (if not the CEO) so that the negative feedback cycle can be broken.

How the “Strong Process” Lever Fails

One step up from project sponsors who will hit the cost lever like a mouse hitting a lever in a skinner box are the project sponsors who put their full faith in processes, such as those proscribed by the PMBOK or recommended by ITIL to solve their cost problems. The thought is that – if only we could get a better plan, and stick to the plan, we could get better at predicting the cost of software.

The chief thing that such frameworks try to do is to do strong change management, contracted service levels, and detailed planning in order to achieve the end of predictable software development. This approach, of course, leads to several problems that are well understood by many people in the Agile and Lean communities but certainly bear repeating:

Software development itself, particularly when done with business analysts, frequently leads to discoveries. As Mary Poppendeick once pointed out, the most valuable feedback for changes to software comes at the latter end of a project, when the group analyzing and developing the software best understands the domain. Traditional processes tend to consider these late changes the most costly, seeking to minimize the highest value work.
Velocity is deeply variable based on factors that management usually does not want to talk about. A good software developer can be 10 times more effective than an average software developer. Other environmental factors that have little to do with the presence of a prescriptive process can cause additional wild swings. A future article in this series will go into more detail about how potential swings in velocity are an order of magnitude more significant than the swings in “cost per hour”.

Proscriptive plans that try to predict cost and schedule without understanding the individuals, service climate, DevOps maturity, and multitude of other variables that can affect the speed of software development will surely fail. The simple fact that corporate budgeting processes select for optimistic thinking most of the time leads to assuming that simple program that consists of 4 main screens will take 8 weeks rather than 8 quarters.

Why is this a cycle of failure? Well – what happens is that you bring PMBOK, ITIL, or some other “capital-P” Process in, and it fails. Because such processes tend to be more accepted by the corporate finance community, the reaction to a failed project tends to be “we didn’t do enough process control”, rather than “we did too much”. The next round of projects tend to have more detailed estimating and stronger change control. There are stories that I personally know of requirements efforts that are measured by years, rather than weeks.

What Levers Remain?

There are levers that go far too much underused. In my experience, they are as follows:

Manage Project Duration Down

In general, prefer smaller projects to larger ones. Predictability works better when the team is smaller and the timeframe is short. I can almost decently predict what one team of 8 people will complete in an iteration if I know all the people, skills, and modes of working on that team. I, personally, am a believer in doing lots of small things rather than one big thing – as it allows for a faster feedback cycle and overall a less risky investment. However, there are many projects – some of the most interesting ones – that will never be small budget, short duration projects. This leads to the lever I use when I can’t manage duration down:

Manage Velocity Up

I have seen a team of 6 people deliver in 7 months in one company the same amount of work I have seen 150 people deliver in 2 years in other companies. And I am not alone, I have talked with others in this business, and nearly everyone I talk to has either seen, or worked on one of these very effective software development teams.

Velocity matters. It is hard to measure productivity the way you measure a person’s shoe size, but you can sure manage relative productivity over a long period of time. The perception that software costs too much is, in my opinion, one of the factors holding back the global economy from expanding. In any large company, any decent developer, would be shocked at the waste involved in many business processes – waste that could be eliminated by better software implementation. There are companies that will not change commission plans, product offerings, or even product pricing to be more profitable, chiefly because of software constraints. If you think software does not have a serious effect on both the opportunities or limitations of a company, you literally must not be paying very much attention.

What’s Next?

In this series of upcoming posts, I am going to explain various factors that affect velocity, and how you can optimize them in a manner that does not sacrifice quality. Factors that, when understood and properly managed, can mean the difference between a flexible company that can respond to market conditions, and a company whose technology that does not allow it to run so much as a second shift when demand ramps up.

Stay tuned…

The Nomadic Developer

Surviving and Thriving in the World of Technology Consulting

Velocity 101: Get the Engineering Practices Right