Is Residuality Theory the Key to Resilient Software Architectures?
While catching up on some conference talks recently, I came across this video from Barry O’Reilly about something I have never heard about before: residuality theory. I recommend checking out the video if you want a quick introduction about the subject directly from the source. I was intrigued, so I bought Barry's book Residues: Time, Change, and Uncertainty in Software Architecture and wanted to share my thoughts on these ideas.
Residuality theory is valuable because it gives architects a structured way to reason about uncertainty and resilience compared to traditional, engineering-focused methods.
An Introduction to Residuality Theory
In a nutshell, Residuality Theory is a method to pressure test an architecture (planned or existing) by brainstorming events that may force the architecture to adapt, then using that information to consider changes to the architecture to make it more resilient.
A simplified analogy is subjecting a newly designed car to real-world testing to see how it stands up to intense heat or cold. The key difference is that software systems are not physical and exist within a complex and ever-changing environment, which requires new terms and techniques. As O'Reilly writes, "How would automobile design look if the properties of rubber, aluminum, asphalt, and gravity were constantly changing?".
Here are a few of those ideas, many of which come from complexity theory and systems thinking:
- A software architecture is a hyperliminal system, meaning it is a complicated (as in a wristwatch) structure living within a complex environment (typically a business) and we fully acknowledge this fact instead of trying to forcefully engineer something using requirements analysis and risk management.
- A stressor is a change event that may force changes to the hyperliminal system for it to survive, which results in a new current state, which is called an attractor. An example of a stressor is one company expanding globally and needing to internationalize its software.
- The naive architecture is what you start with. A residue is what the architecture is after a stressor impacts the system (think
stressor(naiveArchitecture) -> residue). These residues are considered a fundamental unit of architecture in this framework. An example is an order of magnitude increase in order volume on an e-commerce website making it impossible to process credit cards when the order is placed and forcing a change to add a queue and a background job to charge the card some time soon after the order is placed. - The ultimate design goal of an architecture is to achieve criticality, which means the sweet spot between being under-engineered (think monolith) and over-engineered (think microservices). This is measured as a product of three factors: the number of components in a system, the number of connections between them, and the bias of how nodes prefer to connect (for example, if all APIs must be REST). Criticality is a moving target based on the current needs of the hyperliminal system.
That's a lot of concepts to juggle, but applying them is deceptively simple and may be uncomfortable at first for people coming from a software engineering background. Here is the basic process:
First, map the components and data flows in the naive architecture. Then brainstorm lots of stressors. This requires some out-of-the-box thinking. It's not "anything that can go wrong", it's more like "anything outside of your current understanding." Examples could be: a competitor undercuts your business, a new government regulation, geopolitical conflict, cultural norm shift, financial failure of a partner or vendor, and so on. These should be as wild as possible but always relevant to the business and very specific.
Next, analyze each stressor to determine the corresponding residue. Consider capturing the following:
- How that stressor might get detected. It's worth exploring to validate the stressor.
- How that stressor might impact the system and/or the business it operates in. For example, a sudden flood of e-commerce orders may crash the website, resulting in a loss of revenue and negative customer sentiment.
- Identify how the business might choose to react to the situation. For example, they may choose to issue refunds to impacted customers if a system fails. In some cases the business may not be in a position to do anything.
- Identify any technical changes to the system to better handle this situation. For example, you may choose to make the application stateless and add a load balancer for horizontal scaling. In some cases, there may not be any technical changes to make.
Here is an example of a stressor analysis for an e-commerce website:
| ID | Stressor | Detection | Attractor | Business Reaction | Technical Change |
|---|---|---|---|---|---|
| 1 | Viral social media causes a 10x uptick in holiday orders | Alerts, media coverage | Website crashes and cannot process orders | Stateless, horizontal scale, post-order payment processing | |
| 2 | Business hastily expands into Germany | Internal comms | In violation of EU privacy and tax laws, poor UX for German customers | Hire German staff | GDPR changes, VAT, i18n, data sovereignty |
| 3 | Parent company sued for bribing public officials | Press | Public boycott campaign, drop in sales | Hire a public relations / crisis management firm | |
| 4 | Data center outage for five days | Alerts | Lost business and wasted marketing spend | N/A, willing to accept the risk | Multi-homed architecture support |
Finally, create an incidence matrix where you map stressors (rows) against the architectural components (columns) to identify how the different stressors impact the system. A 1 means it's impacted, a 0 means it's not.
| Catalog | Cart | Payment | Fulfillment | Customer Service | TOTAL | |
|---|---|---|---|---|---|---|
| 1. Order spike | 0 | 0 | 1 | 0 | 1 | 2 |
| 2. German expansion | 1 | 1 | 1 | 0 | 1 | 4 |
| 3. Boycott | 0 | 0 | 0 | 0 | 0 | 0 |
| 4. Data center failure | 1 | 1 | 1 | 1 | 1 | 5 |
| TOTAL | 2 | 2 | 3 | 1 | 3 |
How you interpret this table:
- High row totals indicate high cross-cutting impact and a sign that your architecture may be too coupled. In the above example, the data center failure indicates it may be worth designing for hosting the system in more than one location, even if the business doesn't want to actually pay for another data center (currently).
- High column totals indicate components that are the most fragile and/or too coarse-grained. In the above example, it may make sense to separate payment into two things: one for acknowledging a new order, and a separate async process for processing a payment.
- Any time there are multiple 1's in a row is a signal that those components are coupled in some fashion and may be a sign of a missed non-functional requirement.
- Any time you see the same groups of components being affected by different stressors, it's a sign they are tightly coupled and maybe should be combined to simplify the architecture.
- Any time you see many rows with all 0's then you may not be stressing the system enough.
That's the essence of it. Based on this collaborative analysis with stakeholders you can make more informed decisions about how to get closer to an appropriate level of criticality.
You can see it's more of an experimental technique, aimed at poking at the system in a more experience-focused way than the typical static boxes-and-lines approach to architecture.
But We Already Have Agile, YAGNI, Risk Management, and SRE!
This type of analysis may seem necessary when we have other existing approaches to handle uncertainty and resilience. It's a fair point but there are some key distinctions.
Agile is essentially placing smaller bets and assumes the people making today's decisions will be there tomorrow to apply what they learned in a rigorous, accountable, continuously improving fashion. Most agile projects and teams don't work that way, or at least not for very long. The software tends to outlive the agile teams and budgets that made it, which is not a recipe for resilience.
You're Not Going To Need It (YAGNI) is a slogan I've used before. I think it has value in some contexts, such as a flagrant use of technology purely for resume enhancement or novelty. It's also helpful to guide junior engineers as they learn about design patterns to not over-use them. It's good at the code level, but YAGNI is too subjective to be done at the systems level, especially hyperliminal systems that can't really be comprehensively understood.
Risk management is too big of a topic to get into here, but it ultimately relies on likelihood probabilities and coarse-grained analysis that, in most cases, are very biased and subject to manipulation. This is inferior to randomized experimentation of an actual system. Also, it's important to note that there is no such thing today as "brittle software architecture insurance" so risk transfer is not an option.
Site Reliability Engineering (SRE) is a practice made popular by Google where specific roles are empowered to focus on the reliability of systems. O'Reilly touches on techniques like Failure Mode Effect Analysis (FMEA) and Chaos Engineering as techniques in this space that are very valuable and should be done much more often. However, these methods are focused on the current technical state of the system (and typically the infrastructure) in production. Residuality Theory covers the complex environment (economic, political, technical, etc.) around the system in a more experimental way and can be done before a system is deployed.
Why this Resonates
O'Reilly's work resonated with me for a few reasons. I think software architecture is an incredibly difficult, mostly undefined, and increasingly important skill and this technique could go a long way to improve the field.
First, the pace of economic, geopolitical, and technological change is increasing so much that the traditional engineering-based approach to architecture we've all used for so long seems to no longer be honest or adequate. I think the best architects in today's world will be the ones who can think creatively and can handle ambiguity instead of requiring precise requirements and definitions to operate.
Second, resilience is more important than ever to today's established businesses. The COVID pandemic exposed the fragility of lean, global supply chains. Generative AI is making it much easier for new entrants to challenge incumbents. I've personally witnessed several businesses fail to adapt to change because of the under-engineering of their core / differentiated internal systems due to a narrow fixation on controlling costs.
Some people equate resilience to the ability to bounce back from stress. That's important, but I'm referring to the ability for a company to quickly adapt to stress. This is similar to agility, but that's more about placing small bets than it is the survivability of hyperliminal systems.
Third, Residuality Theory puts to words what I've seen the best architects seem to have a knack for. Battle scars from decades of building and supporting systems give senior architects an intuition for how to avoid under-engineering or over-engineering. This "taste" is incredibly valuable, hard to teach, sometimes hard to articulate, hard to screen candidates for, and is by no means perfect. Such skills will be in huge demand given the new economics of software engineering due to LLMs.
By embracing the unknowableness of hyperliminal systems, getting creative, and stressing an architecture through structured experimentation, it may be possible to bake the right amount of resilience into the system that is appropriate for the turbulent business context it lives in.
How to Learn More
I expect more content on this topic from O'Reilly, which seems to be his PhD thesis. If this interests you, definitely check out the book, it's a surprisingly short read. Follow him for more.
After getting your head around the concepts, the best way to learn about this is to try the techniques for yourself. I also don't see why these concepts cannot be applied to things other than software architecture. I may explore that in a future post.
If the theoretical basis behind these ideas interests you, I recommend learning more about systems thinking. This field has a few different academic viewpoints, but essentially focuses on the properties of complex systems and how they evolve over time. A good place to start is Thinking In Systems by Donella Meadows.