The Agile Industrial Complex

30 Jul 2025
11 minute read

Does the flavor really matter?

The Backdrop

We are all sitting there, silently contemplating our existence, in a room that feels more like a massive fish tank than a conference room. Six developers, an intern, and the product owner, a motley mix of expats and locals, when the scrum master pipes up: “What small experiment could we try next sprint to improve our delivery?”. And the response? More silence, awkward at first, then strangely comforting as we are all probably thinking the same thing.

You see, this is the umpteenth sprint where we didn’t deliver on the sprint goal and, according to Jira, the velocity is slowly declining week after week, month after month. The more we push, the less gets done and crucially, the closer we all get to burnout. We’ve tried about everything! Changing the way we do estimates, twice daily standups, more detailed refinements, more refinements, longer sprints, shorter sprints, demo sessions, celebratory moments, longer retrospectives, and about whatever else we could think of.

As the retrospective continues, sticky notes are written, comments are made, fingers pointed at specific issues, processes amended, and off we go to the next meeting, the sprint planning, the cycle continues, the sprints must go on.

For most scrum teams this is an all too familiar scenario. The endless churn of the feature factory with ever expanding backlogs, codes bases full of technical debt, endless meetings, and a team held together by an informal slack channel filled with memes and humor that comes from spending years in the swimlane trenches, lobbing around tickets, pull requests, all the while, juggling support issues and trying to dodge standby duty on christmass.

From up high the struggles are no less real and often existential. With deadlines looming, tight budgets, a permanent talent shortage and the ever present spectre of more efficient competitors putting you out of business. The need for predictability and transparency drives a desperate search for solutions. And what’s the solution employed almost everywhere to solve these crises? Promoted as the one silver bullet that will save us and lead us all to organizational utopia? Agile, with a capital letter A. The framework that promises to solve all our problems while creating entirely new ones.

Ok, let’s take a step back, drop the cynicism and take a look at how we got here.

Software is hard

Getting software development right is hard. It has always been hard. It will always be hard. It’s hard to know how long it will take. It’s hard to know how much it will cost. It’s hard to solve the right problem. Things break, requirements change, people quit, acquisitions happen, deadlines are missed, etc., etc. You get the picture.

Companies and governments have been struggling with this for well over 50 years. There are so many philosophies, tools, frameworks, patterns, models, processes, styles, paradigms and methodologies that it has almost become a field of study on its own.

Some are definitely worse than others but most seem to work to some degree or another. A framework can work well for one team and fail spectacularly for another team within the same organisation due to the differences in the teams. Now consider the variations in organisations. The scope of the problem is staggering and it should be clear that there are no silver bullets, no “one size fits all” schemes.

So what is agile and where did it come from?

You are likely familiar with the origin, so I will keep it short. “Agile” started as nothing more than a label given to a set of values and principles described in a document called the agile manifesto. It was written by 17 software developers at a ski resort in Utah, all frustrated with the heavyweight methodologies dominating the industry at the time.

The manifesto states the following.

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

The 12 principles are:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity–the art of maximising the amount of work not done–is essential.
The best architectures, requirements, and designs emerge from self-organising teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

That’s it. Or is it?

What “Agile” means today

Agile as described in the previous section is so simple and to the point. What changed and why can we barely recognize the term in the previous context with what we see and experience around the corporate world around us every day? Where is the mention of scrum, kanban or even SAFe? A good place to start explaining that is with a quote from 2006. Yes, you read that correctly, 2006. The confusion started more than 20 years ago already and is still going strong.

Semantic diffusion occurs when you have a word that is coined a person or group, often with a pretty good definition, but then gets spread through the wider community in a way that weakens that definition. This weakening risks losing the definition entirely - and with it any usefulness to the term. I’m partly prompted to write this because I see this semantic diffusion happening very publicly to two terms at the moment: ‘agile’ …. -Martin Fowler

Sematic diffusion explains exactly why the term “Agile” is so broadly applied to frameworks outside of the original manifesto. To understand what it has become today, we have to look at how it evolved and eventually permeated the industry so widely.

The industrialization of agile

So what happened between that simple manifesto and the complex world of “Agile” we see today? The answer lies in a fundamental human tendency: when we see something that works, we try to bottle it and sell it.

Some organizations do software development really well. They build products used by millions, handle trillions of requests daily, process terabytes of data without everything falling apart. When they succeed, others desperately want to know: How did they do it?

This is where things get tricky. Success in software development isn’t just about process—it’s about people, culture, communication patterns, organizational structure, technical decisions, market timing, and dozens of other factors that are nearly impossible to measure or replicate. But humans hate uncertainty, especially when millions of dollars are at stake.

So instead of grappling with this complexity, we focus on what we can measure and codify: processes, roles, ceremonies, artifacts. We count story points, track velocity, measure cycle time, and document every step of every workflow. These become the “secret sauce”—the codified processes and procedures that promise to recreate success.

A growing ecosystem of advisory organizations emerged to package these codified processes into sellable frameworks, complete with certifications, training programs, and implementation guides. This became the “Agile Industrial Complex”—an entire industry built around selling what was originally a simple set of values and principles.

The irony is profound: agile started as a rebellion against process-heavy methodologies, yet it spawned an entire industry of new processes, certifications, and frameworks that would make the original waterfall architects proud.

Today, almost every company with an IT department does some flavor of agile. The most prevalent are Scrum and Kanban, though Lean and Extreme Programming and other variants are also popular in some sectors. Interestingly, most of these “Agile” frameworks predated the actual agile manifesto—Scrum from the 1990s, Kanban even older, XP from 1996, and Lean adapted from 1960s manufacturing principles.

So what’s the problem and what are we missing?

How is it that agile is so ubiquitous? Why did so many teams end up choosing these specific set of frameworks? Ideally they would all have done their homework, diligently taken into account their team, organization, culture, goals and, visions, etc, then carefully compared and contrasted the possibilities and finaly tailored a framework to applify their strengths and suplement their weaknesses. Unfortunately A new department is created, a team is formed to execute a project or a new product is set to be developed. To manage the team or project and insure that things run properly, the industry standard is often the go to and applied. A flavor is picked, a professional is hired and off we go.

Scrum promised self-organizing teams and empirical process control. In practice, it often becomes “sprints as mini-waterfalls” where the Scrum Master becomes a project manager in disguise, daily standups turn into status reports, and “sprint commitment” becomes a weapon for deadline pressure.

Kanban promised continuous flow and visual management to optimize delivery. In practice, it often becomes a glorified to-do list where work-in-progress limits are ignored, priorities change daily based on whoever shouts loudest, and “pull-based” work becomes “push whatever the CEO wants” while the board accumulates hundreds of stale tickets that nobody dares to delete.

Lean promised to eliminate waste and maximize customer value through just-in-time delivery. In practice, it often becomes an obsession with removing anything that can’t be immediately justified in a spreadsheet, leading to the elimination of “wasteful” activities like documentation, testing, or architectural planning—until the technical debt becomes so crushing that the team spends more time fighting the system than building features.

Extreme Programming promised high-quality code through engineering practices like pair programming and test-driven development. In practice, it often becomes engineering puritanism where developers spend three days refactoring a function that works perfectly fine, write tests for the sake of coverage metrics rather than actual quality, and dismiss business concerns as “not understanding the technical complexity” while missing every deadline in pursuit of the perfect codebase.

If you’ve spent any time working with development teams in different organizations at all, you will probably have noticed that they usually face similar issues, regardless of which exact flavor of agile they use. This isn’t to say these frameworks are inherently worthless—they all contain valuable insights-the problem is they way the frameworks are used in practice.

An insight from 1967 helps explain why simply adopting a framework or standard method often fails. Conway’s Law states:

organisations which design systems … are constrained to produce designs which are copies of the communication structures of these.

It reveals the fundamental flaw in process-focused transformation: you can’t have cross-functional teams in a company with rigid departmental silos, or self-organizing teams in a command-and-control hierarchy. The process is just the surface layer—the real structure lies beneath in how people actually communicate and make decisions.

But Conway’s Law is hard to sell. It requires organizational change, culture shifts, and addressing uncomfortable truths about power structures. Much easier to sell a framework with clear roles, defined ceremonies, and measurable outputs.

Here is what Martin Folwer has to say about the topic:

I always like to point out that it isn’t methodologies that succeed or fail, it’s teams that succeed or fail. Taking on a process can help a team raise its game, but in the end it’s the team that matters and carries the responsibility to do what works for them. I’m sure that the many Flaccid Scrum projects being run will harm Scrum’s reputation, and probably the broader agile reputation as well. But since I see SemanticDiffusion as an inevitability I’m not unduly alarmed. Teams that fail will probably fail whatever methodology they mis-apply, teams that succeed will build their practices on good ideas and the scrum community’s role is to spread these good ideas around widely. - Martin Fowler

Some companies get it right by embracing agility and some companies get it wrong by implementing “Agile”. The core of agile software development is embracing agility in terms frameworks, processes and methodologies, to keep what works, tweak what kind of works, discard what doesn’t and remember to keep talking and reflecting. Ironically the most “agile” thing you can do is not to do “Agile”.

So what can you do?

Strive to understand. Whether you are a developer, scrum master, delivery manager or just peripherally involved in a software project, it is your responsibility to seek understanding. It is the cornerstone of all meaningful collaboration.

Strive to understand what you are building, but more importantly, why you are building it. What problem are you solving? For whom? What does success look like? Without this shared context, your team is just a feature factory, churning out code without purpose.

Strive to understand how you are building it. This isn’t just about the technical stack; it’s about the trade-offs, the legacy code, the architectural vision (or lack thereof). It’s about knowing why that weird hack was put in place three years ago before you try to rip it out.

And most crucially, strive to understand the people you work with. How does your team actually function, beyond the ceremonies and rituals on the calendar? What are the real communication lines, the unspoken rules, the hidden strengths, and the glaring weak spots? Who thrives in chaos, and who needs structure? Who speaks up, and who needs to be drawn out? When you understand the individuals and the individuals understand each other, then you can begin to understand the team as a system.

When we truly understand—the what, the why, the how, and the who—we can start having the right conversations. We can challenge assumptions, give better feedback, and support each other more effectively. We stop cargo-culting processes and start building a way of working that is tailored to our unique context. Only then can we escape the cycle of failed sprints and meaningless metrics and truly participate.

Only then can we improve our game, regardless of what flavor we chose.