Understanding Misalignments in Software Development Projects

Tags: agile, waterfall, teams, misalignments

Note: This post is a work-in-progress; I expect to iterate over the design and wording of the misalignments to improve clarity and incorporate new ideas and information which may change some aspects.

Misaligning the work we do with how we do it:

When introducing Scrum to your managers and colleagues, you may find yourself in the position of having to convince or "sell" them on the merits of a very different way of working.  While you may all see and feel the outward manifestations of why things haven't been working well like missed deadlines, runaway budgets, sub-par quality, stress and low morale, changing how you work seems like a very risky proposition - especially when weighed against faulting the team, the customer's unrealistic expectations, poorly-written and/or changing requirements and an unsympathetic QA department. The typical management response is to add more process, more management oversight and exhortations for employees to do things "right" or "better".

However, while in the throes of a critical project with money, jobs and credibility on the line, many of us fail to recognize that these "pain-points", contrary to our ingrained cultural beliefs and schooling, are signals that we need to work smarternot harder.  Much like a door that doesn't shut right or a wobbling bicycle wheel that needs truing, there are fundamental misalignments occurring between ourselves, our work, and how we do it that are preventing us from doing better.  They can be ignored, of course, as most organizations do - but this comes at a cost of compensation in other areas.  The result is an organization that is imbalanced and will always struggle to meet expectations.

The aim of this post is to explain six of the most critical misalignments that occur between the nature of the work of software development teams and how they are directed to approach them.  A subsequent post will explore how agile frameworks like Scrum help to address them with simple process changes that bring about often dramatic improvements.

The Six Fundamental Misalignments

A misalignment occurs when something works against its intended purpose or design; this causes side-effects which we interpret as either indications of underlying problems or an intended consequence.  These interpretations take on different characteristics depending on who makes them:  The further from the root causes an individual is, the more likely cognitive biases will interfere with correct diagnosis.  In other words, those who do the work tend to be more in-tune with what's going wrong than those who are directing them.  The caveat to this is when the underlying problem is well-understood.

In traditional software projects, there are six fundamental misalignments that contribute to teams working at cross-purposes, making the goal of delivering valuable software and ROI difficult to almost impossible:

  1. Fallacy of Division of Labour
  2. Defined Process Control Models
  3. People as Replaceable Components (Resources)
  4. Command & Control Team Management
  5. Context Switching/Multi-Tasking
  6. Hero Culture/Worship

To make the misalignments clearer below, I've created a notation that takes the form of {Misalignment} // {Is Misaligned With}:

Fallacy of Division of Labour // All Effort is Design

This first, and most fundamental, misalignment is the consequence of equating how software deveopment projects are planned, organized, staffed and managed with their real-world civil engineering and manufacturing counterparts - specifically the extent to which the effort involved can be effectively partitioned among project team members.

In the case of an engineering project, say building a bridge or skyscraper, labour is divided into two phases, Design and Construction, with highly-skilled engineers, architects and designers drafting plans and instructions in the former for assembly by lesser-skilled trades and labourers in the latter. This division of labour is reasonably cost and schedule efficient because the tasks which constitute the majority of effort, ie. construction, can be well-partitioned among multiple people with little-to-no communication overhead between them.  All that's needed is to follow the plans.  Behind schedule? Add more men.

This paradigm quickly degrades, however, when applied to the context of software development for the simple reason that what constitutes "design" is widely-misinterpreted as "construction". Software is "designed" by human beings using source code which is subsequently interpreted and "constructed" by machines into usable applications and solutions.  The implication of this distinction is that software "construction" is "so cheap as to be free", yet its design so pervasive as to impose significant risk and expense.

We can gain efficiencies of scale by "farming out" the compiling of source code across multiple machines because the rules for doing so are fixed in each compiler and can be run in isolation; we can even do complex pre-processing work across multiple machines to aid in the running of programs, such as the famous Stanford University Folding@Home Distributed Computing Project demonstrates. No such analog, however, exists for designing software - it is intensely intellectual, personal and very difficult to distribute in the same way as a physical task. In fact, as Fred Brooks Jr. noted in The Mythical Man-Month, doing so contributes to lengthening the schedule and thus cost because of the additional communication overhead required between workers to understand how to perform and integrate their work with each other.

An interesting exception to this misalignment is the practice of Pair Programming where two developers work together on a single problem for an undetermined amount of time. This works because the developers are seated next to each other and share a workstation - one who "drives" the keyboard and the other who partners in the problem-solving - thus removing the overhead of communication. Go beyond a pair of developers, however, and problems will follow.

The misalignment of the Fallacy of Division of Labour is vital to understanding the root causes of software project failure because it equates intellectual labour with physical labour, starting a cascade of successive misalignments which I discuss below. I've written on this phenomena in a previous post: The Fallacy of Division of Labour in Software Projects.

Defined Process Control Models // Weakly-Defined or Continuously Variable Inputs

As a consequence of the first misalignment, a second is created that aligns a software project using a defined process control model based on product manufacturing schemes with the intent to produce repeatable outcomes from well-known, defined or fixed inputs.  Most software development projects are not simple enough to have fixed inputs - they continuously vary based on the degree of agreement on requirements with customers/end-users, certainty on the technologies employed and the skill and abilities of the people who will be creating it.  It is a wicked problem that requires an empirical approach based on the principles of visibility, inspection and adaptation at regular, short intervals to tame.

This misalignment also creates another by establishing a "push" model to move artifacts and work products across phases - often completely out-of-synch with the people who need them, eg. too soon, too late or too much all at once. As a consequence, the work system - such as it is - can become stressed and overloaded. This sets the stage for schedule slippages, quality cutting and cost overruns.

I've written about the underlying conditions that create this misalignment in a previous post: No Silver Bullets - Just Hard Work

People as Role-Based, Replaceable Components (Resources) // People as Unique, Skilled Contributors

When a software development project is considered as the outcome of 90% construction effort controlled using a manufacturing-based defined process, the logical extension is to view the people involved as role-based, replaceable components.  This misalignment is made explicit by the repeated-use of the HR terminology of "resource" instead of "contributor", which recognizes that people constitute the most significant variable input to a project.  As a result, organizations "shop" for team members based on the role-based skills they list on their CV instead of looking at their attitudes and abilities.  The resulting frictions create team dysfunctions like heroics, demoralization (why be passionate about what you do when you're just a "Drone in Sector 7-G?"), hiding behind titles ("not my job") and scapegoating.  

It's very difficult to promote true teamwork under these conditions because people are disconnected from their work and an innate desire to be recognized for their contributions. For more insights on this misalignment, see my earlier post on Why Financial Incentives Don't Work which references fantastic examples of people and team dysfunctions by Patrick Lencioni and Dan Pink.

Command & Control Team Management // People as Self-Organizing, Rational, Skilled Adults

As a result of viewing people as replaceable components and not unique contributors, the next misalignment  of using command and control team management arises.  This comes from a mindset/cognitive bias that only through direct and persistent management oversight, intervention and coercion can organizations be led. At the employee level, this has an infantilizing and degrading effect, building into people the expectation that they will be told what to do instead of being trusted to organize themselves around and be responsible for their work. Worse: It actively discourages the opportunities for creative thinking by making it explicitly risky to do so.  Who wants to lose their job over trying something new that might not work? Overall, this misalignment is about trust and the issues managers have with giving control over to those who are better-suited to solving on-the-ground problems.

While an unfortunate reality for many, the good news is that this misalignment is undergoing corrections across multiple industries, including software. For example, see how the highly-successful video game company, Valve, operates without any direct management oversight. Also see Ralph Stayer's excellent 1985 essay, How I Learned to Let My Workers Lead, for an example of how a CEO transformed his business to let workers set the pace and direction of change.  Also see this recent CNN article that outlines a shift in the thinking of business leaders to connect and collaborate over command and control.

When managers begin to view their people as people who are rational, intelligent adults capable of independent decision-making, a new style of management emerges, which Takeuchi and Nonaka observed in their 1986 Harvard Business Review article, The New New Product Development Game, as subtle management. I'll expand on this in a subsequent post.

Context Switching/Multi-Tasking // Individual and Team Effectiveness

When we think of people as replaceable components in a fixed system, it's not uncommon for an additional misalignment to occur where we view their individual capacities to respond to ad-hoc or multiple requests as nearly infinite.  This is called "multi-tasking" or "context switching" and is rooted in a mythology that equates the practice with being highly-effective. In reality, multi-tasking impairs our ability to focus and see any task through to completion. 

In a 2009 Stanford University study of high and low multi-taskers, the hypothesis of high-multitasking effectiveness was tested with not-so-surprising findings:  Those who identified themselves as high-multitaskers underperformed those who were low-multitaskers or single-taskers in response to multiple and varied stimuli. Why?

Part of the explanation resides in how the human brain works:  When we are called upon to switch between tasks in different problem domains (eg. projects) we need to mentually "unload" one to "load" or "reload" another.  This takes time and is impacted by things like fatigue and distractions. This is why we feel slightly confused when we switch contexts between different problems and begin to grasp for mental guideposts to re-establish our original train of thought. This effect can be countered by taking reminder notes of your thought processes, but this incurs additional overhead.

When the effects of context switching and multi-tasking are multiplied across teams and their organization, the impact of this misalignment becomes evident in the stress it creates, causing interpersonal relationships to degrade as employees become overloaded.  Context Switching has become an unfortunate, false badge of honour in the workplace: Its practice is an indication of an organizational culture that places a premium on capacity over throughput.

Hero Culture/Worship // Clarity/Constancy of Purpose

A direct consequence of the aforementioned misalignments is an organization that is in a perpetual struggle to keep itself together in spite of itself. As a result, it is seen that only "heroes" can help save contracts and steward departments toward profitability. Thus we see absurd job ads that make the call for "rockstars", "ninjas", "samurais" and a host of other mythological warriors to come to the rescue - for a great reward.

This culture of hero-worshipping is a misalignment with having individuals and organizations not having clarity and constancy of purpose.

Clarity of Purpose is a skill as much as a mindset - it speaks to understanding the context of a problem domain, ie. why it is being taken on and for what purpose. If an organization's ethos is to "push" work on to teams and provide little context, it is demonstrating incoherent purpose and will likely suffer a host of side-effects such as team demoralization and an emphasis on bureaucracy over entrepreneurialship and true innovation.

In their book, Personal KanbanJim Benson and Tonianne DeMaria Barry note from their experience that the "A-List" developers who demonstrated clarity of purpose on their software teams were ones who "took the time to learn why they were building the software in the first place. They sought clarity from the onset, gathering vital information and incorporating it into their design."

Constancy of Purpose is a closely-related concept that W. Edwards Deming described in his landmark book, Out of the Crisisas the first of 14 Points of Effective Management:

Create constancy of purpose toward improvement of product and service, with the aim to become competitive, stay in business and to provide jobs.

When we allow a culture of hero-worshipping to govern our hiring and reward systems, we are attempting to superficially solve deeper problems in the work we do and how we do it which keeps us from gaining constancy of purpose. Instead of seeking ways to continuously improve our delivery, we instead look toward HR practices, organizational bureaucracy and hierarchies to boost our performance - with mediocre-at-best results. Any gains we might make are illusions as they disguise the effectiveness of the above-mentioned misalignments, excusing them as the failings of our people rather than the system they have to work within.  It is no surprise, therefore, that hero-worshipping organizations have the most difficulty letting this misalignment go when transitioning toward superior team-based collaboration.

(See my earlier post on Deming's Red Bead Experiment for a practical demonstration on the ineffectiveness of focusing on people over their system of work.)

Almost Inextricably Out-of-Alignment

In their recent book, Software in 30 DaysKen Schwaber and Jeff Sutherland remark that, "[businesses] have been ill-served by the software industry for forty years - not purposefully, but inextricably". This is the consequence of introducing layers of cascading and inter-related misalignments into organizations and projects between how software teams work and the work they do. As a result, the organization works against and in spite of itself, often in the dark about the fundamental missteps that are occurring, and often exacerbating the situation by inappropriately treating symptoms.

A critical tipping point is reached when the symptoms of the misalignments begin to overtake the ability of people within the organization and/or their customers to tolerate them. It is at this time that an exploration for new ways to manage the work becomes the imperative. Recognizing and understanding the six fundamental misalignments I've outlined above will enable business owners, managers and team members to begin to deconstruct their mindsets, work habits and philiosophies toward their work and start to introduce corrections to bring them back into alignment for greater effectiveness.  This is the key to understanding the raison d'etre for almost all agile software development frameworks, including Scrum.

In a subsequent post I will explore how simple changes in behaviour works to realign teams with their purpose of creating valuable software (and thus ROI) for their customers.

blog comments powered by Disqus