The Reasoning Behind Software Processes

While serving as Scrum Master for much of last year, I had more than a few mini-existential crisis-es. Not the run-of-the-mill nihilistic ones that have served as a near-keynote to our 21st century, but ones surrounding the purpose of process in software. I put hours-upon-hours into this role, and yet I had nothing concrete to show for it. No working software made by “Scrum”, and no proof that Scrum has helped my team deliver value. Though incredibly rewarding, my role didn’t provide the direct feedback I’ve come to expect as a developer. It was hard to know which actions worked, and which didn’t. What was too much, and what wasn’t enough. Where I was helping the team by providing more clarity & feedback, and where I was hurting by dragging them into boring meetings, the same ones I dreaded for years as a developer.

After a few years pondering this, I developed a basic philosophy for software process application. It’s simple & crude, probably too crude. It’ll evolve as I mature in this industry, and may be proven outright false. But it hasn’t let me down so far:

• Developers are amazing beings who, given a clear goal and the time, space, & resources needed can solve almost any problem
• While a one-person project can be easily understood, prioritized, and built from the ground-up without any overhead, things get complicated when you add more people & scope
• As scope & headcount grows, priorities, goals, & the overall vision becomes muddied
• Some sort of process is required to point a team (or teams) of people in the right direction, making sure everybody knows what needs to be done, why they’re doing it, and when it needs to be done by
• Process also provides space & perspective to monitor & correct practices. As scope & headcount grows, it becomes tough to understand what’s happening at a higher-level, and tougher to fix systematic, team-wide problems. Processes can raise a red flag when indicators exceed boundaries, and may provide guidance for removing bottlenecks & valueless work
• Process should not exceed these three objectives of Direction, Monitoring, & Correction, and should always be applied in service of the team, not the other way around. Teams must first invest in the vision, and understand the direction of their project, before voluntarily taking on appropriate process for the sake of delivering greater value and providing an overall better, more useful/engaging/delightful piece of software. Processes that are carelessly selected for a team just produce apathy, dissatisfaction, and waste
• Software processes should ultimately enable the first bullet point, by providing team members with the direction, tools, & environment needed to flex their natural talents for the sake of delivering the maximum amount of value possible

Process Anti-Patterns

Too Much Process

There’s nothing like meetings about what meetings you should have, or meetings that produce no value except to follow a chapter in the “15 Steps to Managing Enterprise Software Projects” book, to make you feel like you live in a Dilbert-esque corporate hellhole where the only escape after 40 years of tedious BS is an ill-attended retirement party and 20 years at the end of your life to watch Wheel-of-Fortune re-runs, but I digress.

To some people a large amount of process is comforting. You get to hold a lot of meetings, play buzzword-bingo through long email chains, and work long hours without actually producing anything that can be scrutinized, criticized, or rejected.

But users don’t use your meetings. They don’t interact with your email chains. They don’t gain value from your perfect execution of every meeting & artifact specified by your process of choice. Instead they use, interact with, and gain value from your software; a large ball of ideas, opinions, and choices comprised of code and assembled by people with wildly different skills, interests, talents, and backgrounds.

Process is the four-on-the-floor beat of your disco song (I’m old on the inside, don’t mock me). It’s not flashy, and certainly not the focal point, but it ties everything together, and provides the rhythm, direction, and continuity needed for flashy guitar solos, epic vocal harmonizations, and kick-a** features to be built and thrive.

Too Little Process

On the flip side, a good-sized team with no process & few rituals can lead to chaos. As mentioned above, process provides tools & insights to help teams prioritize what’s important, define what they should do, complete work in a fluid manner, and occasionally self-analyze to determine what aspects of their day-to-day practices are not working & need to change.

I’ve seen it first-hand, where a team with no defined process built services that were never used, while critically-required services weren’t known about until it was almost too late. The simple question of “what should we do next?”, was answered differently by every stakeholder, and the result was a team that either whiplashed from requirement-to-requirement, never fully finishing anything, or put all their focus on work that delivered little value.

For this team the simple act of gathering all stakeholders in a room, writing down their high-level requirements, and prioritizing them was a godsend. Developers now knew not only what should be done, but why on earth we’re doing it. This enabled a dialog between them & stakeholders to flesh out exact requirements & come up with better solutions. Through estimation, defined sprints, and sprint planning we could provide more accurate, reasonable answers when stakeholders asked “when will this be done by?”, and work with those stakeholders to ensure that the most critical work was being done first. Through retrospectives developers were able to speak up about what hampered them, worried them, excited them, or frustrated them, and provide management with guidance for how to better serve it’s team.

Corporations tend to lean towards process-heavy side, which is why developers are often dragged into useless meetings & pointless conversations. But too little process can also frustrate developers. We all like to know what we should do, why we should do it, and when it should be done by. Process can help to answer those questions for larger teams in complicated domains.

The Wrong Process

A professor at my college had a framed, gold paint-dipped hammer on his office’s wall to prove a point. That there is no perfect tool, paradigm, or process that applies to every software project.

Scrum has been the “hot thing” for at least a decade now, and when applied well it’s been very effective in smart, dynamic teams whose requirements can be defined 2 weeks in advance, but not necessarily 2 months in advance. Teams whose time & budget are fixed, but whose exact product feature list has some flexibility. Teams that can work autonomously, without constantly depending on external teams for critical needs. A team can tell when Scrum would be a good fit for them, based on their current situation & constraints.

Unfortunately not every organization defers the process decision to it’s teams, instead requiring a single process for all scenarios. This can lead to a bastardized form of the process, where teams have to bend over backwards to satisfy it’s minimal requirements, causing waste & inefficiencies.

Process should be selected based on the environment, team structure, purpose, and constraints of the project. If requirements are chiseled into rock day-one (example: a direct port of a current system, no changes or updates to functionality) and time/budget are flexible, then waterfall is the way to go. If requirements are highly variable and the team’s tech-stack/structure supports effective flow & CI/CD (Continuous Integration/Continuous Delivery), then Kanban may be the solution to get new features in front of users quicker than every two to four weeks. Let the context of the project drive process selection, and you’ll spend less time fighting your process, and more time using it more effectively put smiles on the faces of your stakeholders.

Non-Adapting Process

In the raging debate of whether you should bend the rules of your process, I’m very much in the camp of flexing where needed, when it’s applicable to your project. Though the creators of modern-day software processes are very smart people, they’re not gods. The rules of Scrum weren’t carved upon stone tablets at the top of Mt. Sinai for us mere mortals to interpret as pure & immutable truth, they were created by a group of developers who were fed up with projects running over scope/budget/schedule.

Note that there are two reasons why you would flex your process.

1. Because it’s hard to satisfy what’s asked of the team
2. Because it doesn’t fix the context around & variables in your project

#2 is a valid reason for modifying your process. #1 is not.

Processes can be so much more than just a way to manage a group of developers. They can provide the goals & tools needed to improve the team, forcing them to eliminate waste & find better ways to deliver awesome work.

Finishing features in 2 weeks (like in scrum) is a lot like trying to run 10 miles. The first time you try you’re going to fail, and you’re probably going to fail dramatically. You’re out of shape, and untrained. But even though you didn’t hit your goal, your body got a little leaner, faster, and stronger. You keep trying, stopping your run at 1, 2, 4, 6, 7, 8 miles, gradually getting farther each day as your fitness improves. Finally you reach the mythical 10 miles, and celebrate the achievement with a good beer. Looking back, 10 miles wasn’t the point. It’s just an arbitrary number you picked from the proverbial dartboard. The point was that you had something to measure your progress against. Something to shoot for. Something to “harden your edge” against as your fitness improved.

Processes can work the same way. Difficult challenges like the short iteration times in scrum can seem impossible at first, and you’ll probably fail at meeting expectations for a while. But that failure, so long as it’s not cast in a defeatist light, sharpens the team’s capabilities. With guidance (or on their own) the team starts to improve communication between members, understand their capabilities, and trim what’s holding them back from their goal. Skills develop, bonds form, and team gelling occurs. Finally the team begins to consistently deliver high-quality, finished work within the set time period at a pace sustainable to them. The iteration period of two weeks was arbitrary. I could have been 4, 3, 1, 6, etc. What’s important is that the team had a mark to shoot for, and by trying, failing, & learning they gradually improved their delivery capabilities, until meeting their goal became second-nature.

So if your process doesn’t perfectly fit your team & situation, and you’ve given your team time to grow against it’s requirements, then change it. But don’t be so quick on removing process requirements just because it’s hard.