Agile testing: Importance of a testing strategy

As you know from our history we didn't have much (if any) systematic testing going on, when we decided to introduce Scrum back in 2009.

A history of no testing equates to several challenges:

• The system is (was) not very well tested
• The codebase and architecture is (was) not very testable (it hasn't been written to be tested)
• The developers are (were) not used to having to care much about "testing"

It was important for our success that the strategy was clear from the beginning, from a management perspective. Testing had to be made important, it needed to be lifted up from the bottom to the top of the priority list. And, as they say, the proof is in the pudding. From a management perspective such as strategy has to be backed up by the attitude of allowing quality assurance to take time.

Our aim from the beginning was the following:

• We should introduce the Tester role in the team and clearly define its responsibilities (certainly harder than it sounds, more on this topic later).
• We should immediately stop taking new shortcuts (i.e. stop increasing our Technical debt)
• Test early and test often. All functionality implemented during a sprint should be tested and, if needed, corrected during the same sprint; while still having a reasonably short sprint length (i.e. two weeks maximum)
• Every user story that we implemented and delivered in every sprint had to be tested
• Create a minimal set of test cases that we could start using to verify the fundamental functionality of the system
• Be pragmatic and realize that we can't go the entire distance in the first sprint. We had to accept that despite our intentions we wouldn't have a truly releaseable product increment after every iteration. There would be critical bugs that we wouldn't have time to fix, there would be too few test cases to cover all functionality, etc. But striving towards the goal was a great start, and it meant we got better and better with every sprint.

Since we wanted to include testing in every sprint (and each sprint was just 1-2 weeks long) we had to figure out a way to expand test coverage without spending unreasonable amounts of effort. The solution (of course) was to automate testing. More on that in coming articles... :-)

Agile testing: Our background

I usually refer to the group of people I manage as "my team", but in fact we are several teams today.

When the development of the product first started back in 2004, as a prototyping/proof-of-concept effort, the "team" then consisted of only 1-2 developers and a very visionary leader (not me :-)). As the prototype became more and more potent and the market showed a positive response, the team was allowed to grow but it was still, in many ways, in a "prototyping/proof-of-concept" mode.

Up until 2008/2009 there was very little systematic testing going on. The development didn't follow any particular development model or strategy. But on the other hand the developers were very skilled, highly motivated and committed, passionate even, largely I think because they had a lot of influence on what went into the product back then. They all pretty much had their own specialist area and their own agendas in terms of what "cool stuff" they wanted to work on and put into the product.

And that was all ok, most of it was intentional, and it obviously worked well. In fact, it was a necessity, I think, to have that way of working in order to achieve what was achieved back then; to quickly get a proof-of-concept out there to check if the idea worked and if there was a market for it, and build it from there.

However. As the team size grew it became increasingly obvious that some structure was needed. As more and more customers installed the product the need for control from a product development perspective grew too; i.e. the need to be able to decide exactly what features the team should work on and when. Likewise, the need for maintaining project overview grew. And so did the need for testing. The result (i.e. the quality and contents of a release) needed to become controlled and predictable. Also, it was desired that the team should adopt development processes and practices used in other parts of the company - and hence no longer be allowed to work in such a chaotic unplanned/uncontrolled manner. It was time to align the product, the team and its way of working with the rest of the company.

It was at this point that I joined, early 2009, and we immediately started introducing Scrum and the Agile philosophy. There had been some efforts to introduce Scrum ("Scrumish") in other parts of the company but there were no success stories, and nobody had really tried to implement Scrum "all the way".

To give you an idea of what the situation was like before the Scrum introduction, here are a few characteristics (end of 2008):

• Relatively unorganized development efforts
• High entrepreneurial spirit in the group
• Plenty of shortcuts taken (Technical debt created)
• Unclear project control/steering
• Highly motivated, committed, self-organizing individuals
• High degree of freedom and innovation spirit
• Very little degree of cooperation among engineers
• Direct contact with customers
• No structured testing
• No "tester" role existed
• Definitely no automated testing
• No process improvement efforts
• Aging codebase, with a continuously increasing technical debt
• No requirement or test specifications
• The work didn't follow the established policies and processes within the rest of the company

And here is that wishlist, i.e. what we wanted to achieve:

• Maintain same or achieve better efficiency
• Maintain same or achieve better level of commitment and creativity
• Maintain same sense of freedom and influence
• Possibility for a "product manager" to be responsible for, and in control of, the direction & content of the product
• Reliable and repeatable delivery result
• Possibility for maintaining project overview
• Stop increasing the technical debt (no more shortcuts)
• Partial refactoring of the codebase/architecture to improve/remove some of the prototype code that remained
• Introduce systematic testing of some kind
• Use the resources available in the company's QA department
• Customer Support department to handle direct customer contacts
• Start following the established processes and policies within the rest of the company

Thankfully, largely due to good, flexible, patient and somewhat daring managers (such as Head of QA, Head of our department, the Head of R&D and also Product Management) my team was allowed to deviate somewhat from the established (largely waterfall-based) processes in favor for a "true" Scrum implementation.

Now, almost three years down the line, we've come far. The team is much bigger and we (think we) work in an agile way and have more or less fully adopted the Scrum methodology.

Here are a few characteristics of how we work today (2011) so you get an idea of what changes we've struggled with:

• We are now 4 cross-functional Scrum teams working in parallel on the same product
• All team members sit together
• 5-7 persons in every team, including 1-2 testers
• Still highly motivated and committed team members; both the "old" and the new
• Highly self organizing teams with a high degree of and natural focus on cooperation
• 1 Scrum Master (might not be optimal now that we've become as many as 4 teams)
• 1 Scrum Product Owner
• Each iteration is 9 work-days long (starts on a Thursday and ends on a Tuesday)
• Estimation is done in Story points
• We measure Velocity and we keep a Release plan up to date, and base time commitments on, measured velocity
• High degree of test automation; 1350 automated function (blackbox) tests, 100 automated "GUI tests"/"end-to-end tests", 2534 automated unit tests and around 110 manual test cases
• Small working (completed) product increments are delivered every iteration
• Our goal is to be continuously releaseable (truly releaseable after every sprint) but we are not there yet
• Natural and continuous focus on process improvement by every team and team member
• Continuous and reoccurring discussions about efficiency and quality
• We have a strategy (and resources) for handling "old" bugs found during sprints
• We have a strategy (and resources) for replacing some of the manual tests with automated tests
• We have a strategy (and resources) for writing automated unit tests for old (legacy) units and not just new ones

A lot of work remains and I'm sure we wouldn't pass any Scrum Compliance Test (even if there was one) with a 100% score, but we've come a long way compared to where we were only two-three years ago. And besides; we haven't adopted Scrum for the sake of adopting Scrum.

In following articles I'll describe and elaborate on some of the experience and lessons I think we've stumbled across during this journey, in particular lessons related to testing and quality assurance (as this article series focuses on Agile Testing :-)).

Agile testing: Start of article series

Wow. It's sure been quiet on my part for a while now. I guess that is what happens when you have kids; priorities in life change. With a 1.5 year old at home and a full-time job I've found myself having virtually no time to spare to write these posts about Agile development and Scrum.

However, I was recently asked by a colleague to come to the Q3 meeting of SAST Öresund (Swedish Association for Software Testing: http://www.sast.se) and hold a 45-minute presentation on the topic of Agile Testing (under the theme "Agile methods in practise"). This was the first time in a while that I've had the chance, and the challenge, to really focus my thoughts around an agile topic and had do an in-depth analysis of my own opinions and experiences.

The presentation was quite interesting and great fun to do, and from what I could tell the interest from the audience was quite high.

So. In the name of openness I figured that I'd share on this blog the topics I brought up in the presentation. I'll post this as an article series and do a bit more elaboration on each topic than what is possible in a brief concentrated 1-hour seminar.

So keep on popping by this blog from time to time, as I'll be posting articles in this series here as often as I have time.

And as always, feel free to drop me a comment with your opinions and own experiences, it's always interesting to hear what other people think.

How to fail with a Scrum introduction

The journey of introducing Scrum in an organization is a huge topic in itself, and nowadays there are loads of books and articles describing different aspects of this. Tens of thousands of companies have successfully adopted an Agile methodology. “Scrum” is no longer just a buzzword as it was a few years ago. The hype, the initial excitement and thereby, hopefully, some of the overconfidence has settled a bit. At least that is the feeling I get when speaking to managers and colleagues “out there”. One would think, then, that the process of introducing Scrum would be easier today. With all that collective experience in the industry, with all those books and articles.


But looking around at all those companies and organizations who claim they have adopted (or tried to adopt) Scrum, I wonder how many of them are/were really doing Scrum “by the book”? How many of them modified Scrum beyond what could still reasonably, with a clean conscience, be called Scrum – and struggled and had problems with what they thought was Scrum because of it?


It doesn’t really matter in the successful cases, where the modifications to Scrum work and the result is something that is at least as good as whatever was being done previously. I guess those organizations would still, in the end, claim that their Scrum implementation was successful – regardless of it really is Scrum or not. But what about all those cases where an organization tried Scrum, concluded it didn’t work, and maybe even decided to go back to dysfunctional..err..sorry, traditional, waterfall-based approach? How many of those ever stopped and pondered on the reasons – the root causes – of why Scrum didn’t work? Ever heard something like “Well, Scrum just didn’t suit our organization/business/team, I think Scrum is over-rated and as I thought it wasn’t the magic bullet answer to everything”?


Scrum and Agile is so simple and made up mostly of common sense behaviors and values, that if it really doesn’t work for you then you’re really special. And no offence but it is likely that you’re pretty much like everyone else – no matter how special you want to believe you are.


So why doesn’t Scrum work?


The person responsible for driving the Scrum introduction – the “Scrum driver” – is key to the outcome of the change. In my experience there are four (five) common traps that a Scrum adopting organization risk falling into. All relate to the Scrum driver:

1. He or she lacks sufficient knowledge about what the Scrum methodology and the Agile philosophy is all about.
2. He or she is also a Project Manager (or a group of Project Managers).
3. He or she lacks enough time to focus on things like coaching, monitoring, adjusting, educating and motivating the Scrum teams and the stakeholders.
4. There is no plan on how to introduce Scrum.

Maybe a fifth one should be listed also, for the record; the failure to realize that there is need for a “Scrum driver”.


Lack of conviction


I think a common reason for failing with Scrum is that the introduction is driven by someone who lacks conviction and (true) passion – i.e. someone who consequently lacks credibility in the eyes of all stakeholders. Introducing Scrum requires a firm and honest belief in its benefits. The need for conviction is not unique to a Scrum driver but apply to any change process. Introducing Scrum will mean drastic changes for everyone; team members, project managers, stakeholders and managers surrounding the development organization/team. The ways of thinking, the methods, attitudes and expectations will need to change. From day one. You can’t introduce Scrum in steps. It’s All in – or leave the table. And people are naturally conservative. It’s always easier to stick to what you know and have experience with. The unknown is scary. Especially at times of stress, when the going gets tough. That’s when conviction and passion will be required in order for the changes – the Scrum methodology – to persevere.


A Project Manager is put in charge


A Project Manager ought to be the one with the best understanding of how to run a software development project. Right? They are (mostly) educated, intelligent, communicative and multi-tasking individuals with loads of real-life experience of stressful situations, risk management and change management. They are used to following checklists, processes and project methodologies. If anyone should be able to introduce Scrum it would be a Project Manager. Right? Well. Would you let a taxi driver be responsible for introducing free subways? Silly analogy perhaps, but my point is that it would be fair to expect their level of commitment to be less than maximal.


Don’t get me wrong, some people are less conservative than others. Open-mindedness is a property that some people have more of – and some less. Introducing Scrum will without a doubt affect the every-day work of the Project Manager, their entire role and responsibility. It may even eliminate the need for a Project Manager altogether. Therefore, to me at least, it is simply not natural that the project manager by default becomes the person responsible for driving that change. They are an involved and affected party. Someone else should be in the driver’s seat, someone like a line manager, a manager of the project office, whoever is the project manager’s boss, or even someone external to the team.


Underestimating the challenge


Introducing Scrum isn’t free – and I’m not talking about a few thousand bucks to get someone a Scrum Master certification, or a couple of books. The adopting organization will need to put the money where its mouth is. In my mind it is not enough to only educate whoever becomes Scrum Master. The entire team needs to have a good (or great) understanding of Scrum; about what problems it solves, what are Agile attitudes (and what are not), what parts of Scrum are fundamental and about what parts can be tailored to the organization/business/team (and which can’t). Educating an entire team will cost time and money.


And education is only the beginning. Using Scrum will require lots of monitoring, coaching and adjustments along the way. Continuous improvement. Inspect and adapt. At first that will be managed by the person driving the Scrum implementation, and therefore will require a lot of time & effort from that person to stay close to the team, to keep up-to-date – by the hour – about what’s going on in the team and outside. But as the Scrum team grows used to the new way of working they will start helping each other out, correcting each other and helping each other with staying within the boundaries of Scrum.


Another aspect often underestimated may be that of getting everyone onboard. Getting the team members to accept Scrum is a pretty obvious challenge that has to be dealt with. The same should be true for the project manager(s), and maybe even the customer(s). But what about surrounding managers, departments and other roles and neighbors within the organization? Those are the people that will be indirectly affected by the Scrum introduction. It is important to have a good idea about who those persons are, so that you can actively work on their expectations. Take control. If surrounding expectations are not in-line with what Scrum is capable of you’re in trouble. That’s why it’s a good idea to form a plan on how to get them to accept Scrum.


Lack of strategy, lack of plan


It’s surprising (disturbing), I think, how little planning is generally made for introducing Scrum – considering that it happens in an environment where people are usually very used to making rigorous plans, managing risks, etc.
A sufficient “plan” in this case would mean taking the time to think about things like;

• Why are we introducing Scrum? What are the expected benefits? Are they realistic?
• What are the expected drawbacks of introducing Scrum? Will all stakeholders accept those?
• Who will be in charge of the Scrum introduction, i.e. who will be the “Scrum driver”? Is the person in charge also a Project Manager?
• Which persons will be directly affected by the change? Is the Scrum driver one of them?
• Which persons will be indirectly affected? Is the Scrum driver one of them?
• What is the level of credibility, passion and conviction of the Scrum driver? Is he/she able and ready to be a source of inspiration to others during the change process?
• Are all affected persons (directly and indirectly) onboard?
• Do we have any key people to get onboard as assisting informal Scrum advocates within the organization/business/team?
• What is our education plan (plan for getting people on the bus) for people directly as well as indirectly affected by the Scrum introduction?
• What is the budget for “education” (getting people on the bus), in people-time or in money, for the persons directly affected by the change?
• What is the budget for “education” (getting people on the bus), in people-time or in money, for the persons indirectly affected by the change?

Summary


Introducing Scrum is a challenge. But I’m pretty sure most would agree that it’s worth the effort, if you can get it to work for you. But please don’t underestimate the challenge. And please consider letting someone other than a Project Manager be in charge of the Scrum introduction.


Whoever is put in charge of the Scrum introduction it needs to be someone with:

1. mandate and authority to implement the change; and
2. someone with a wholehearted belief in the benefits of the change; and
3. someone with a budget for the extra effort caused by the Scrum introduction.

Let me be clear on this: I don’t think that Scrum really is the solution to everything. I’m sure there are many cases and circumstances where it simply doesn’t suit the organization/business/team. But if you tried it and failed, I think it’s worth to ponder on whether or not it was Scrum that really failed – or the implementation of it.


As always, I’m interested in getting some feedback via comments on this blog.


Cheers.

Overcommitment is better than undercommittment

This time I want to talk about the amount of user stories that a scrum team decide they feel comfortable committing to at the beginning of a sprint (at spring planning) - and what happens when they're approach the end of the sprint.

At sprint planning, have you ever had a discussion with (or in) your team about whether or not to include that last story or not in the sprint backlog? The one that they're unsure they'll be able to complete. What did you/they decide, and why? Did you hear anyone say something like "Well, let's stick with these fewer stories - we can always include that other one later during the sprint, if we should find ourselves with time to spare"?

If the team undercommits, i.e. includes a set of user stories that sums up to a lesser scope (a smaller sprint backlog), it means that they're reasonably likely to complete all stories committed to - and at the end of the sprint they have a good chance of getting a feeling of accomplishment and success. They delivered all the stories! Yay! And, as indicated by the quote above, they can always add that extra story at the end of the sprint if they complete the committed sprint backlog early. Right?

Constant overcommitment - on the other hand - might cause a feeling of failure, as the team never really manage to deliver what they commit to.

But, to be honest, how often has your team found themselves with time to spare at the end of a sprint? And how many times have they included that extra story from the top of the backlog at the end of the sprint? Somehow, the time available in a sprint always seem to be just enough for exactly (or less than) those stories committed to at the sprint planning - rarely more.

While I think we need to recognize and respect the potential negative consequences of constant failure by constant overcommittment, I think we also need to remember that people often need (and like) a healthy amount of stress/pressure in order for them not risk producing waste by extra effort (refer to Lean Software Development for more information about extra effort and waste).

The Agile approach is of course to try both alternatives a couple of times, and then inspect and adapt. Find out what suits your team(s). In my personal preference, however, I think reasonable overcommitment carries greater benefits than drawbacks.

Either way, the team owns the decision. It's their commitment. I for one want to remind my teams about and get them thinking about this for themselves. Keep it in mind at sprint planning.

There is a name of this phenomenon: Parkinson's Law; "Work expands so as to fill the time available for its completion" (http://en.wikipedia.org/wiki/Parkinson's_law).

Another approach used by some, is to not expect the team to include more stories during a sprint even if they finish early. As a general rule. The thinking behind that practice is that the team's commitment at the beginning of a sprint is what they agree on with the scrum product owner. And since that's what the scrum product owner expects, that's what is enough to deliver. Any spare time at the end of the sprint is used for other things; anything from free time (off work), lab activities, or whatever other fun things you could think of.

I think this approach is excellent in theory, and it sure sounds nice; if you finish early, you can go home. However as a general rule, in practice, when a deadline is approaching and you want to get as far as you can down the product backlog, i doubt that it would be possible to stick to this. Let me know if this works for you. I'm curious!

And as always, please give me some feedback. Let me know how you think about and handle this.