Agile testing: Our background

I usually refer to the group of people I manage as "my team", but in fact we are several teams today.

When the development of the product first started back in 2004, as a prototyping/proof-of-concept effort, the "team" then consisted of only 1-2 developers and a very visionary leader (not me :-)). As the prototype became more and more potent and the market showed a positive response, the team was allowed to grow but it was still, in many ways, in a "prototyping/proof-of-concept" mode.

Up until 2008/2009 there was very little systematic testing going on. The development didn't follow any particular development model or strategy. But on the other hand the developers were very skilled, highly motivated and committed, passionate even, largely I think because they had a lot of influence on what went into the product back then. They all pretty much had their own specialist area and their own agendas in terms of what "cool stuff" they wanted to work on and put into the product.

And that was all ok, most of it was intentional, and it obviously worked well. In fact, it was a necessity, I think, to have that way of working in order to achieve what was achieved back then; to quickly get a proof-of-concept out there to check if the idea worked and if there was a market for it, and build it from there.

However. As the team size grew it became increasingly obvious that some structure was needed. As more and more customers installed the product the need for control from a product development perspective grew too; i.e. the need to be able to decide exactly what features the team should work on and when. Likewise, the need for maintaining project overview grew. And so did the need for testing. The result (i.e. the quality and contents of a release) needed to become controlled and predictable. Also, it was desired that the team should adopt development processes and practices used in other parts of the company - and hence no longer be allowed to work in such a chaotic unplanned/uncontrolled manner. It was time to align the product, the team and its way of working with the rest of the company.

It was at this point that I joined, early 2009, and we immediately started introducing Scrum and the Agile philosophy. There had been some efforts to introduce Scrum ("Scrumish") in other parts of the company but there were no success stories, and nobody had really tried to implement Scrum "all the way".

To give you an idea of what the situation was like before the Scrum introduction, here are a few characteristics (end of 2008):

• Relatively unorganized development efforts
• High entrepreneurial spirit in the group
• Plenty of shortcuts taken (Technical debt created)
• Unclear project control/steering
• Highly motivated, committed, self-organizing individuals
• High degree of freedom and innovation spirit
• Very little degree of cooperation among engineers
• Direct contact with customers
• No structured testing
• No "tester" role existed
• Definitely no automated testing
• No process improvement efforts
• Aging codebase, with a continuously increasing technical debt
• No requirement or test specifications
• The work didn't follow the established policies and processes within the rest of the company

And here is that wishlist, i.e. what we wanted to achieve:

• Maintain same or achieve better efficiency
• Maintain same or achieve better level of commitment and creativity
• Maintain same sense of freedom and influence
• Possibility for a "product manager" to be responsible for, and in control of, the direction & content of the product
• Reliable and repeatable delivery result
• Possibility for maintaining project overview
• Stop increasing the technical debt (no more shortcuts)
• Partial refactoring of the codebase/architecture to improve/remove some of the prototype code that remained
• Introduce systematic testing of some kind
• Use the resources available in the company's QA department
• Customer Support department to handle direct customer contacts
• Start following the established processes and policies within the rest of the company

Thankfully, largely due to good, flexible, patient and somewhat daring managers (such as Head of QA, Head of our department, the Head of R&D and also Product Management) my team was allowed to deviate somewhat from the established (largely waterfall-based) processes in favor for a "true" Scrum implementation.

Now, almost three years down the line, we've come far. The team is much bigger and we (think we) work in an agile way and have more or less fully adopted the Scrum methodology.

Here are a few characteristics of how we work today (2011) so you get an idea of what changes we've struggled with:

• We are now 4 cross-functional Scrum teams working in parallel on the same product
• All team members sit together
• 5-7 persons in every team, including 1-2 testers
• Still highly motivated and committed team members; both the "old" and the new
• Highly self organizing teams with a high degree of and natural focus on cooperation
• 1 Scrum Master (might not be optimal now that we've become as many as 4 teams)
• 1 Scrum Product Owner
• Each iteration is 9 work-days long (starts on a Thursday and ends on a Tuesday)
• Estimation is done in Story points
• We measure Velocity and we keep a Release plan up to date, and base time commitments on, measured velocity
• High degree of test automation; 1350 automated function (blackbox) tests, 100 automated "GUI tests"/"end-to-end tests", 2534 automated unit tests and around 110 manual test cases
• Small working (completed) product increments are delivered every iteration
• Our goal is to be continuously releaseable (truly releaseable after every sprint) but we are not there yet
• Natural and continuous focus on process improvement by every team and team member
• Continuous and reoccurring discussions about efficiency and quality
• We have a strategy (and resources) for handling "old" bugs found during sprints
• We have a strategy (and resources) for replacing some of the manual tests with automated tests
• We have a strategy (and resources) for writing automated unit tests for old (legacy) units and not just new ones

A lot of work remains and I'm sure we wouldn't pass any Scrum Compliance Test (even if there was one) with a 100% score, but we've come a long way compared to where we were only two-three years ago. And besides; we haven't adopted Scrum for the sake of adopting Scrum.

In following articles I'll describe and elaborate on some of the experience and lessons I think we've stumbled across during this journey, in particular lessons related to testing and quality assurance (as this article series focuses on Agile Testing :-)).