The "Sleeping beauty"-effect in agile software architecture

Agile projects, above all those with multiple Scrum teams, need a moderated architecture process, in order to facilitate architectural decisions. It is advantageous to decide as late as possible, nevertheless the correct point in time should be chosen intentionally. If not, important architecture decisions could be overlooked or made too late.

The promise

In the ideal agile world, architecture grows by evolution (“no big upfront design”). An architecture will not be sketched out with great detail before development starts, but it will be created instead step by step during the work in the sprints. Decisions will be taken when triggered by work on a current user story. Until then, the team has time to gather know-how which will come in handy for evaluating architectural decisions. Since the “big upfront design” doesn’t have to be adjusted along the way, requirements can be accepted more easily und can be integrated into development more quickly. As a consequence, risk will be lower and efforts will decrease – that’s the hope, at least. Even the quality should increase, because decisions can be made with more experience. All this is based on the implicit assumption, that all knowledge needed to take the decision will be accessible for all persons involved at the correct point in time. 

The reality

This assumption is – as we will see – incorrect, which leads to some negative effects. Some preliminary work on the architecture already helps with the beginning of an agile project. The question, which part of the system the scrum team should work on can only be addressed sensibly, when there is a rough breakdown of the “system”.

Basically, it’s not so bad an idea to decide late. In fact, there are many architecture questions which can be postponed. E.g. you don’t need to decide on a browser, when you don’t work with browser specific functions. You don’t need to talk about implementation details of microservices, which will be accessed via webservice API. To decide late indeed means to decide with greater knowledge and therefore, to decide with greater quality. Thus, the focus can be shifted to the current and immediate problems.

Picture 1: Sleeping beauty phase

Yet, this also is the catch: the “immediate” problems tend to push the postponed decisions from the agenda. They enter a “sleeping beauty” mode, and waking them up is hard. Somewhere in the project wiki they are documented, but who keeps track? All of a sudden the team faces the problem, that a new component has to be implemented in the next sprint – but the decision about its interior technology has not been taken. Important business information and conceptual information is missing, which is necessary to make the decision, but which does take some time to acquire. As we see, the assumption of the “always available information” is not true.

Due to the „sleeping beauty“ effect, the project gets late and misses its schedule – which could be prevented by some preliminary work. The same effect also casts its shadow on the hopes for better quality, since decisions taken under time pressure tend to be suboptimal. Furthermore, non-functional requirements also fall prey to that effect, which can lead to even more ugly consequences.

Best practices

But there is an effective method against the „sleeping beauty“-effect: somebody has to watch the “sleeping” decisions and has to ensure, that necessary knowledge is acquire early enough. To that end, architecture decisions must be moderated with regard to all teams on the project. The moderating instance has to take care for postponed decisions and is responsible to get them up on the agenda right on time. That’s early enough, to carry out important preliminary work, e.g. to clarify business relevant questions. Some decisions even shouldn’t be postponed at all (e.g. because there are dependencies to other topics). This weighing-up should also be done by the architecture instance (you might call it a “guild”).

Sometimes, there are dependencies between various decisions, which may even apply to different teams at first sight. Those overarching dependencies must be identified, tracked and resolved, in order to achieve overall consistency. Experience shows, that it’s a good idea to transfer that responsibility to a dedicated group of people. This enables evolutionary development of the architecture without taking the risk of the “sleeping beauty” effect.

The importance of Software Architecture in Agile projects

Last week I gave a presentation on agile project management. One of the talk's main points is, that software architecture and documentation are very important even in agile projects. That’s contrary to popular belief since many people seem to follow Dilbert’s paradigm “no planning and no documentation – just start hacking and complaining”.

My argument refers to the accumulation of technical and business related knowledge in the project. The project begins with a minimum of knowledge, and while the teams works, develops and releases it also gains new insights and new wisdom about their technical and their business domain. At the end of the project’s duration they end up with a maximum of knowledge. Conversely, the risk of making wrong decisions (based on false or deceivingly true knowledge) decreases with time (s. figure 1). That is the case with nearly every non-trivial software project.  

Unfortunately, in traditional software projects mostly all of the decisions are made in the early phases of the project, when knowledge still is insufficiently low (s. figure 1). That's why early decisions run a high risk of being wrong. Consequently, most of these projects are doomed to run into trouble. 

In agile projects on the other hand, decisions are distributed more evenly over the projects timeline. As the saying goes, they are deferred to the “latest responsibly possible moment”. Thus, we have decisions in all stages of the projects (s. figure 2). Early decisions need anticipation, since they always involve some guesswork about what e.g. the customer might expect or what the future market might look like. Later decisions need adaptation, because they can alter earlier decisions or even involve some remaking of the software.

Therefore, both methods – anticipation as well as adaptation – are needed to manage projects, and especially in agile projects. Both have to be in some balance (s. figure 3). But – and this is my main point – in order to enable the team to anticipate and to adapt, the project needs a solid base, which can be established only by correct documentation, sound software architecture and low technical debt.

E.g. software architecture is needed for both: anticipation and adaptation. If you anticipate some future demand (let’s say, lots of data) you may decide to include a bigger and more scalable database. This is a anticipating architecture decision. Likewise, if you have to adapt and must change something in the software, you’d desperately need a stable architecture, because if it’s not, your software might break through the change, or the change might not be possible at all. Thus, we can see that software architecture is at least as important in agile projects as in traditional, if not even more important.