Scrum FTW - Scrum for the win!

Agile testing: Importance of a testing strategy

2011-10-03T10:21:00.011+02:00

As you know from our history we didn't have much (if any) systematic testing going on, when we decided to introduce Scrum back in 2009.

A history of no testing equates to several challenges:

The system is (was) not very well tested
The codebase and architecture is (was) not very testable (it hasn't been written to be tested)
The developers are (were) not used to having to care much about "testing"

It was important for our success that the strategy was clear from the beginning, from a management perspective. Testing had to be made important, it needed to be lifted up from the bottom to the top of the priority list. And, as they say, the proof is in the pudding. From a management perspective such as strategy has to be backed up by the attitude of allowing quality assurance to take time.

Our aim from the beginning was the following:

We should introduce the Tester role in the team and clearly define its responsibilities (certainly harder than it sounds, more on this topic later).
We should immediately stop taking new shortcuts (i.e. stop increasing our Technical debt)
Test early and test often. All functionality implemented during a sprint should be tested and, if needed, corrected during the same sprint; while still having a reasonably short sprint length (i.e. two weeks maximum)
Every user story that we implemented and delivered in every sprint had to be tested
Create a minimal set of test cases that we could start using to verify the fundamental functionality of the system
Be pragmatic and realize that we can't go the entire distance in the first sprint. We had to accept that despite our intentions we wouldn't have a truly releaseable product increment after every iteration. There would be critical bugs that we wouldn't have time to fix, there would be too few test cases to cover all functionality, etc. But striving towards the goal was a great start, and it meant we got better and better with every sprint.

Since we wanted to include testing in every sprint (and each sprint was just 1-2 weeks long) we had to figure out a way to expand test coverage without spending unreasonable amounts of effort. The solution (of course) was to automate testing. More on that in coming articles... :-)

Agile testing: Our background

2011-09-30T13:52:00.025+02:00

I usually refer to the group of people I manage as "my team", but in fact we are several teams today.

When the development of the product first started back in 2004, as a prototyping/proof-of-concept effort, the "team" then consisted of only 1-2 developers and a very visionary leader (not me :-)). As the prototype became more and more potent and the market showed a positive response, the team was allowed to grow but it was still, in many ways, in a "prototyping/proof-of-concept" mode.

Up until 2008/2009 there was very little systematic testing going on. The development didn't follow any particular development model or strategy. But on the other hand the developers were very skilled, highly motivated and committed, passionate even, largely I think because they had a lot of influence on what went into the product back then. They all pretty much had their own specialist area and their own agendas in terms of what "cool stuff" they wanted to work on and put into the product.

And that was all ok, most of it was intentional, and it obviously worked well. In fact, it was a necessity, I think, to have that way of working in order to achieve what was achieved back then; to quickly get a proof-of-concept out there to check if the idea worked and if there was a market for it, and build it from there.

However. As the team size grew it became increasingly obvious that some structure was needed. As more and more customers installed the product the need for control from a product development perspective grew too; i.e. the need to be able to decide exactly what features the team should work on and when. Likewise, the need for maintaining project overview grew. And so did the need for testing. The result (i.e. the quality and contents of a release) needed to become controlled and predictable. Also, it was desired that the team should adopt development processes and practices used in other parts of the company - and hence no longer be allowed to work in such a chaotic unplanned/uncontrolled manner. It was time to align the product, the team and its way of working with the rest of the company.

It was at this point that I joined, early 2009, and we immediately started introducing Scrum and the Agile philosophy. There had been some efforts to introduce Scrum ("Scrumish") in other parts of the company but there were no success stories, and nobody had really tried to implement Scrum "all the way".

To give you an idea of what the situation was like before the Scrum introduction, here are a few characteristics (end of 2008):

Relatively unorganized development efforts
High entrepreneurial spirit in the group
Plenty of shortcuts taken (Technical debt created)
Unclear project control/steering
Highly motivated, committed, self-organizing individuals
High degree of freedom and innovation spirit
Very little degree of cooperation among engineers
Direct contact with customers
No structured testing
No "tester" role existed
Definitely no automated testing
No process improvement efforts
Aging codebase, with a continuously increasing technical debt
No requirement or test specifications
The work didn't follow the established policies and processes within the rest of the company

And here is that wishlist, i.e. what we wanted to achieve:

Maintain same or achieve better efficiency
Maintain same or achieve better level of commitment and creativity
Maintain same sense of freedom and influence
Possibility for a "product manager" to be responsible for, and in control of, the direction & content of the product
Reliable and repeatable delivery result
Possibility for maintaining project overview
Stop increasing the technical debt (no more shortcuts)
Partial refactoring of the codebase/architecture to improve/remove some of the prototype code that remained
Introduce systematic testing of some kind
Use the resources available in the company's QA department
Customer Support department to handle direct customer contacts
Start following the established processes and policies within the rest of the company

Thankfully, largely due to good, flexible, patient and somewhat daring managers (such as Head of QA, Head of our department, the Head of R&D and also Product Management) my team was allowed to deviate somewhat from the established (largely waterfall-based) processes in favor for a "true" Scrum implementation.

Now, almost three years down the line, we've come far. The team is much bigger and we (think we) work in an agile way and have more or less fully adopted the Scrum methodology.

Here are a few characteristics of how we work today (2011) so you get an idea of what changes we've struggled with:

We are now 4 cross-functional Scrum teams working in parallel on the same product
All team members sit together
5-7 persons in every team, including 1-2 testers
Still highly motivated and committed team members; both the "old" and the new
Highly self organizing teams with a high degree of and natural focus on cooperation
1 Scrum Master (might not be optimal now that we've become as many as 4 teams)
1 Scrum Product Owner
Each iteration is 9 work-days long (starts on a Thursday and ends on a Tuesday)
Estimation is done in Story points
We measure Velocity and we keep a Release plan up to date, and base time commitments on, measured velocity
High degree of test automation; 1350 automated function (blackbox) tests, 100 automated "GUI tests"/"end-to-end tests", 2534 automated unit tests and around 110 manual test cases
Small working (completed) product increments are delivered every iteration
Our goal is to be continuously releaseable (truly releaseable after every sprint) but we are not there yet
Natural and continuous focus on process improvement by every team and team member
Continuous and reoccurring discussions about efficiency and quality
We have a strategy (and resources) for handling "old" bugs found during sprints
We have a strategy (and resources) for replacing some of the manual tests with automated tests
We have a strategy (and resources) for writing automated unit tests for old (legacy) units and not just new ones

A lot of work remains and I'm sure we wouldn't pass any Scrum Compliance Test (even if there was one) with a 100% score, but we've come a long way compared to where we were only two-three years ago. And besides; we haven't adopted Scrum for the sake of adopting Scrum.

In following articles I'll describe and elaborate on some of the experience and lessons I think we've stumbled across during this journey, in particular lessons related to testing and quality assurance (as this article series focuses on Agile Testing :-)).

Agile testing: Start of article series

2011-09-30T11:35:00.004+02:00

Wow. It's sure been quiet on my part for a while now. I guess that is what happens when you have kids; priorities in life change. With a 1.5 year old at home and a full-time job I've found myself having virtually no time to spare to write these posts about Agile development and Scrum.

However, I was recently asked by a colleague to come to the Q3 meeting of SAST Öresund (Swedish Association for Software Testing: http://www.sast.se) and hold a 45-minute presentation on the topic of Agile Testing (under the theme "Agile methods in practise"). This was the first time in a while that I've had the chance, and the challenge, to really focus my thoughts around an agile topic and had do an in-depth analysis of my own opinions and experiences.

The presentation was quite interesting and great fun to do, and from what I could tell the interest from the audience was quite high.

So. In the name of openness I figured that I'd share on this blog the topics I brought up in the presentation. I'll post this as an article series and do a bit more elaboration on each topic than what is possible in a brief concentrated 1-hour seminar.

So keep on popping by this blog from time to time, as I'll be posting articles in this series here as often as I have time.

And as always, feel free to drop me a comment with your opinions and own experiences, it's always interesting to hear what other people think.

How to fail with a Scrum introduction

2011-03-02T16:37:00.007+01:00

The journey of introducing Scrum in an organization is a huge topic in itself, and nowadays there are loads of books and articles describing different aspects of this. Tens of thousands of companies have successfully adopted an Agile methodology. “Scrum” is no longer just a buzzword as it was a few years ago. The hype, the initial excitement and thereby, hopefully, some of the overconfidence has settled a bit. At least that is the feeling I get when speaking to managers and colleagues “out there”. One would think, then, that the process of introducing Scrum would be easier today. With all that collective experience in the industry, with all those books and articles.

But looking around at all those companies and organizations who claim they have adopted (or tried to adopt) Scrum, I wonder how many of them are/were really doing Scrum “by the book”? How many of them modified Scrum beyond what could still reasonably, with a clean conscience, be called Scrum – and struggled and had problems with what they thought was Scrum because of it?

It doesn’t really matter in the successful cases, where the modifications to Scrum work and the result is something that is at least as good as whatever was being done previously. I guess those organizations would still, in the end, claim that their Scrum implementation was successful – regardless of it really is Scrum or not. But what about all those cases where an organization tried Scrum, concluded it didn’t work, and maybe even decided to go back to dysfunctional..err..sorry, traditional, waterfall-based approach? How many of those ever stopped and pondered on the reasons – the root causes – of why Scrum didn’t work? Ever heard something like “Well, Scrum just didn’t suit our organization/business/team, I think Scrum is over-rated and as I thought it wasn’t the magic bullet answer to everything”?

Scrum and Agile is so simple and made up mostly of common sense behaviors and values, that if it really doesn’t work for you then you’re really special. And no offence but it is likely that you’re pretty much like everyone else – no matter how special you want to believe you are.

So why doesn’t Scrum work?

The person responsible for driving the Scrum introduction – the “Scrum driver” – is key to the outcome of the change. In my experience there are four (five) common traps that a Scrum adopting organization risk falling into. All relate to the Scrum driver:

He or she lacks sufficient knowledge about what the Scrum methodology and the Agile philosophy is all about.
He or she is also a Project Manager (or a group of Project Managers).
He or she lacks enough time to focus on things like coaching, monitoring, adjusting, educating and motivating the Scrum teams and the stakeholders.
There is no plan on how to introduce Scrum.

Maybe a fifth one should be listed also, for the record; the failure to realize that there is need for a “Scrum driver”.

Lack of conviction

I think a common reason for failing with Scrum is that the introduction is driven by someone who lacks conviction and (true) passion – i.e. someone who consequently lacks credibility in the eyes of all stakeholders. Introducing Scrum requires a firm and honest belief in its benefits. The need for conviction is not unique to a Scrum driver but apply to any change process. Introducing Scrum will mean drastic changes for everyone; team members, project managers, stakeholders and managers surrounding the development organization/team. The ways of thinking, the methods, attitudes and expectations will need to change. From day one. You can’t introduce Scrum in steps. It’s All in – or leave the table. And people are naturally conservative. It’s always easier to stick to what you know and have experience with. The unknown is scary. Especially at times of stress, when the going gets tough. That’s when conviction and passion will be required in order for the changes – the Scrum methodology – to persevere.

A Project Manager is put in charge

A Project Manager ought to be the one with the best understanding of how to run a software development project. Right? They are (mostly) educated, intelligent, communicative and multi-tasking individuals with loads of real-life experience of stressful situations, risk management and change management. They are used to following checklists, processes and project methodologies. If anyone should be able to introduce Scrum it would be a Project Manager. Right? Well. Would you let a taxi driver be responsible for introducing free subways? Silly analogy perhaps, but my point is that it would be fair to expect their level of commitment to be less than maximal.

Don’t get me wrong, some people are less conservative than others. Open-mindedness is a property that some people have more of – and some less. Introducing Scrum will without a doubt affect the every-day work of the Project Manager, their entire role and responsibility. It may even eliminate the need for a Project Manager altogether. Therefore, to me at least, it is simply not natural that the project manager by default becomes the person responsible for driving that change. They are an involved and affected party. Someone else should be in the driver’s seat, someone like a line manager, a manager of the project office, whoever is the project manager’s boss, or even someone external to the team.

Underestimating the challenge

Introducing Scrum isn’t free – and I’m not talking about a few thousand bucks to get someone a Scrum Master certification, or a couple of books. The adopting organization will need to put the money where its mouth is. In my mind it is not enough to only educate whoever becomes Scrum Master. The entire team needs to have a good (or great) understanding of Scrum; about what problems it solves, what are Agile attitudes (and what are not), what parts of Scrum are fundamental and about what parts can be tailored to the organization/business/team (and which can’t). Educating an entire team will cost time and money.

And education is only the beginning. Using Scrum will require lots of monitoring, coaching and adjustments along the way. Continuous improvement. Inspect and adapt. At first that will be managed by the person driving the Scrum implementation, and therefore will require a lot of time & effort from that person to stay close to the team, to keep up-to-date – by the hour – about what’s going on in the team and outside. But as the Scrum team grows used to the new way of working they will start helping each other out, correcting each other and helping each other with staying within the boundaries of Scrum.

Another aspect often underestimated may be that of getting everyone onboard. Getting the team members to accept Scrum is a pretty obvious challenge that has to be dealt with. The same should be true for the project manager(s), and maybe even the customer(s). But what about surrounding managers, departments and other roles and neighbors within the organization? Those are the people that will be indirectly affected by the Scrum introduction. It is important to have a good idea about who those persons are, so that you can actively work on their expectations. Take control. If surrounding expectations are not in-line with what Scrum is capable of you’re in trouble. That’s why it’s a good idea to form a plan on how to get them to accept Scrum.

Lack of strategy, lack of plan

It’s surprising (disturbing), I think, how little planning is generally made for introducing Scrum – considering that it happens in an environment where people are usually very used to making rigorous plans, managing risks, etc.
A sufficient “plan” in this case would mean taking the time to think about things like;

Why are we introducing Scrum? What are the expected benefits? Are they realistic?
What are the expected drawbacks of introducing Scrum? Will all stakeholders accept those?
Who will be in charge of the Scrum introduction, i.e. who will be the “Scrum driver”? Is the person in charge also a Project Manager?
Which persons will be directly affected by the change? Is the Scrum driver one of them?
Which persons will be indirectly affected? Is the Scrum driver one of them?
What is the level of credibility, passion and conviction of the Scrum driver? Is he/she able and ready to be a source of inspiration to others during the change process?
Are all affected persons (directly and indirectly) onboard?
Do we have any key people to get onboard as assisting informal Scrum advocates within the organization/business/team?
What is our education plan (plan for getting people on the bus) for people directly as well as indirectly affected by the Scrum introduction?
What is the budget for “education” (getting people on the bus), in people-time or in money, for the persons directly affected by the change?
What is the budget for “education” (getting people on the bus), in people-time or in money, for the persons indirectly affected by the change?

Summary

Introducing Scrum is a challenge. But I’m pretty sure most would agree that it’s worth the effort, if you can get it to work for you. But please don’t underestimate the challenge. And please consider letting someone other than a Project Manager be in charge of the Scrum introduction.

Whoever is put in charge of the Scrum introduction it needs to be someone with:

mandate and authority to implement the change; and
someone with a wholehearted belief in the benefits of the change; and
someone with a budget for the extra effort caused by the Scrum introduction.

Let me be clear on this: I don’t think that Scrum really is the solution to everything. I’m sure there are many cases and circumstances where it simply doesn’t suit the organization/business/team. But if you tried it and failed, I think it’s worth to ponder on whether or not it was Scrum that really failed – or the implementation of it.

As always, I’m interested in getting some feedback via comments on this blog.

Cheers.

Overcommitment is better than undercommittment

2011-02-10T14:22:00.006+01:00

This time I want to talk about the amount of user stories that a scrum team decide they feel comfortable committing to at the beginning of a sprint (at spring planning) - and what happens when they're approach the end of the sprint.

At sprint planning, have you ever had a discussion with (or in) your team about whether or not to include that last story or not in the sprint backlog? The one that they're unsure they'll be able to complete. What did you/they decide, and why? Did you hear anyone say something like "Well, let's stick with these fewer stories - we can always include that other one later during the sprint, if we should find ourselves with time to spare"?

If the team undercommits, i.e. includes a set of user stories that sums up to a lesser scope (a smaller sprint backlog), it means that they're reasonably likely to complete all stories committed to - and at the end of the sprint they have a good chance of getting a feeling of accomplishment and success. They delivered all the stories! Yay! And, as indicated by the quote above, they can always add that extra story at the end of the sprint if they complete the committed sprint backlog early. Right?

Constant overcommitment - on the other hand - might cause a feeling of failure, as the team never really manage to deliver what they commit to.

But, to be honest, how often has your team found themselves with time to spare at the end of a sprint? And how many times have they included that extra story from the top of the backlog at the end of the sprint? Somehow, the time available in a sprint always seem to be just enough for exactly (or less than) those stories committed to at the sprint planning - rarely more.

While I think we need to recognize and respect the potential negative consequences of constant failure by constant overcommittment, I think we also need to remember that people often need (and like) a healthy amount of stress/pressure in order for them not risk producing waste by extra effort (refer to Lean Software Development for more information about extra effort and waste).

The Agile approach is of course to try both alternatives a couple of times, and then inspect and adapt. Find out what suits your team(s). In my personal preference, however, I think reasonable overcommitment carries greater benefits than drawbacks.

Either way, the team owns the decision. It's their commitment. I for one want to remind my teams about and get them thinking about this for themselves. Keep it in mind at sprint planning.

There is a name of this phenomenon: Parkinson's Law; "Work expands so as to fill the time available for its completion" (http://en.wikipedia.org/wiki/Parkinson's_law).

Another approach used by some, is to not expect the team to include more stories during a sprint even if they finish early. As a general rule. The thinking behind that practice is that the team's commitment at the beginning of a sprint is what they agree on with the scrum product owner. And since that's what the scrum product owner expects, that's what is enough to deliver. Any spare time at the end of the sprint is used for other things; anything from free time (off work), lab activities, or whatever other fun things you could think of.

I think this approach is excellent in theory, and it sure sounds nice; if you finish early, you can go home. However as a general rule, in practice, when a deadline is approaching and you want to get as far as you can down the product backlog, i doubt that it would be possible to stick to this. Let me know if this works for you. I'm curious!

And as always, please give me some feedback. Let me know how you think about and handle this.

Back in business - with questions about Test and Development

2010-11-02T07:45:00.000+01:00

Long time no see!

I've been silent for a while now, due to the extremely premature birth of my daughter this spring (she was born over 3 months early). Everything is OK now and she (Emelie, is her name by the way) is doing well. It was, not surprisingly, very chaotic times there for a while, and add to that getting used to being a parent which in itself is a big change.

Anyway, for these reasons I haven't really been focused on working with Scrum, but lately, as I'm getting back in to the groove of things here at work, I find myself thinking more and more about Why's and Why-Not's with regard to Scrum & Agile software development.

My team(s) here have expanded during the year. They're now a total of about 15 engineers (about 1/4 testers and 3/4 software developers). Which brings me to the topic of this post; how do other organizations cope with running Scrum in a traditional organization where there's one department for "Software development" and another for "Software Quality Assurance"?

I've managed to bring the Test Engineers into the team by means of physically locating them here where the Software Developers sit (instead of having the testers sit on another floor). Very cross-functional. Excellent! But, that was the easy part, now to the trickier bit; how do you get the team to think of themselves as one unit with a joint mission (and thereby sharing tasks without particular regard to their formal role) rather than one group of Software Developers and one group of Test Engineers? I often have to remind members of the team (more often Software Developers than Test Engineers, to be honest) that tasks such as "Write automatic test" or "Execute test case A" doesnt automatically have to be done by a Test Engineer just because the task description includes the word "test" - it may just as well be done by a Software Developer.

As you can probably guess I am of the firm opinion that every task on the Scrum Board should be up for grabs by anyone in the team (at least that should be the default), regardless of if says "Software Developer" or a "Test Engineer" on the individual's employment contract. They're all comitted to achieving the sprint goal and it's thereby in everyone's interest that every task is completed.

It all sounds good and is easy enough to grasp in theory, but in reality it's harder to achieve. Partly (I think) because this is a traditional organization where people are employed specifically as "Test Engineer" and "Software Developer". People tend to kling to that, and I sometimes have a hard time getting them to step outside the boundaries of their formal/traditional role. A complicating factor is that it is not naturally bidirectional; a Software Developer usually have no real problem of carrying out a task that says "Test" on it, but a Test Engineer is not as often able to carry out a task that says "Program" on it.
Does this support the notion not to try to share tasks across roles?

Well. I like to think of the role "Test Engineer" as a person with a (usually :-)) formal education and experience in the field of "software quality assurance" and that those guys and girls are those who are skilled in identify testing challenges, quality risks and generally ensure that what they (the Scrum team) build is tested "enough". But "Test Engineer" doesnt automatically mean "Those are the guys and girls who should carry out all the tasks with the word 'Test' written on them".

To sum this up; If it was completely up to me I would get rid of the formal roles and just hire "Scrum Team Members" with different backgrounds (some who are more skilled in programming and some who are more skilled in the field of software verification & validation).

As always, I like getting feedback and comments from you about this and other posts.

See you soon(er)!

It's all about priorities

2010-03-21T18:57:00.004+01:00

I've been pretty silent on my Scrum blog lately (well, for the past few months even). The main reason is that we've been expecting a child and we've had a relatively problematic pregnancy; the climax of which led up to my beautiful daughter Emelie's very premature birth on the 17th of March 2010.

She was born in week 26 + 3 days only, so she is 3 months premature. She was 1230 grams, 38cm tall, when she came out. She entered the world at 09:57 in the morning, with three minutes to spare to my team's Daily Scrum!! :-)

Kidding aside, me and my girlfriend Helene are now living at the hospital in Lund, tending to little Emelie's needs and helping her grow and get stronger.

I'm currently on leave from work and I will continue to be somewhat out of the Scrum & Agile loop for some time ahead. We're taking it day-by-day here.

Such are priorities in life.

I keep a blog about my daughter's progress. You can find it here: Liten Emelie (sorry, only in Swedish - she doesn't know English just yet ;-)).

New version of Backlog Manager

2010-03-10T10:40:00.003+01:00

Recently I've had some time to spare, so I decided to refactor my Backlog Manager into a new version.

Here it is: Backlog Manager v10.1

It is more robust than the previous versions, and has better support for Release Planning.

Moved/updated Excel tool links

2010-02-18T13:59:00.003+01:00

Due to some technical issues the Excel tools that I've made has been unavailable for a week or so. The files have now been moved to a new temporary location.

Here are new links:

Backlog Manager
Velocity Log
Estimation Sheet

What unit to use in burndown?

2009-11-27T08:01:00.001+01:00

What unit do you use for plotting your Burndown Chart? Do you use “Remaining Hours” of the tasks? Or perhaps “Story Points” of the stories in the sprint backlog? Or something else?

Up until recently I’ve taught Scrum teams to just sum up the number of remaining hours for all tasks on their Scrum Board, and plot that sum on the burndown chart. That has been my default approach and I haven’t really experimented with it.

Why even bother breaking down a Story into Tasks?
The immediate answer to this question is of course that we do it to be able to sum up estimated Remaining Time. Duh :-). But I think there’s more to it, because a Story is more than just the sum of its Tasks. The real reason, I believe, is to get the team thinking; to get the team to reflect over what challenges the Story hides so that they maintain an as clear picture as possible of where the implementing is heading and how the sprint is progressing. This implies that breakdown of a story shouldn’t be done just once (e.g. at sprint planning). The view of what tasks a story contains changes as the implementation progresses and things are discovered. It’s natural that the contents of the story vary over time. Nothing of what I have written here is controversial. It’s all part of Scrum.

What happens in practice?
Plotting a burndown based on the sum of Remaining Time obviously requires that the team first breaks down the Stories into Tasks. This, in my experience, has always been somewhat of a challenge; for the team to actually end up with useful tasks. A good breakdown of a scope into User Stories is hard enough, and then to also successfully break a Story down into good Tasks … Well, it is hard and it requires lots of practice. I’ve always struggled getting teams to continuously reflect over their breakdown and about how to improve it. But in the end, tasks often seem to end up saying more or less meaningless things like “Implementation”, “Testing” and “Review”. Fail.

Failure in Task breakdown like this effectively prohibits the team from having any use of it. A breakdown like this doesn’t say much about the Story and consequently there is little or no use in continuously reviewing the breakdown to try to learn more about the Story as implementation progresses. The only remaining purpose of the breakdown, then, is to force a daily re-estimation, so that numbers can be summed up and plotted on the chart. The burndown and the estimation of Remaining Time becomes an end in itself.

We started counting "Number of remaining tasks"
My team, being in the situation described above, felt that the daily task re-estimation was (more or less) meaningless, and proposed that they should try and completely skip the estimation all together. If nothing else, it would save them that estimation-time that was borderline waste anyway. But they also realized that the burndown has an important purpose (yesterday’s weather, indicating progress, early indication of problems, etc) so they felt that they still needed burndown. But if they don’t estimate Remaining Time, then what do they plot on the burndown chart? Well, they agreed to try to plot the number of remaining (not done) Tasks. Simple! And not completely new; it’s a fairly common practice, if I’m not mistaken.

Interesting consequences of this change
This change in the process had a very interesting side-effect: all of a sudden the Task breakdown produced a large number of highly meaningful tasks. Instead of just saying “Implementation” the tasks that the team now produce actually describe what needs to be done, in pretty deep detail. The number of tasks produced both at the sprint planning and on the Standups is significantly higher now than before, and they say a lot more about what needs to be done.
It just seem to come more naturally now. The only difference is that we no longer estimate the tasks afterwards. That’s it. I think this is very interesting. It’s like if there was a mental block before, that has now been removed once the team doesnt have to bother estimating the tasks. I don’t know, but whatever the reason, I think it's great. The team is satisfied with the change. I feel that the quality of the task breakdown of the Sprint backlog is much higher now, and consequently the understanding of the team’s progress is much better (both for themselves and for me as a stakeholder). The project is more transparent.

The drawback of this change might be that the burndown probably is a little less accurate, because the size of tasks vary a lot. On the other hand, how accurate was the burndowns before, when we were estimating remaining time almost as an end in itself? Hopefully the accuracy now is not much worse. And if it is, I think it is outweighed by the gain in insight and increased feeling of progress.

Let me know how you work with tasks, and why… As always I’m curious to get some feedback.

Agile projects are never late

2009-11-20T17:18:00.004+01:00

In the post I want to talk alittle about "late projects". It is a topic that is somewhat current for me and I have spent some time pondering about this phenomenon :-).

The question I ask is: What does it mean for a project to “be late”?

Well, in my view, in order to “be late” there has to be a target-date or a target-state (e.g. deadline or a certain scope or both) that the project’s current state is compared against, and the result of the comparison is that the state is such that either expected date or expected scope isn’t achievable.

If we can agree that this would be some reasonably valid definition of “being late” then it implies that the following factors are involved in “becoming late”:
• Date
• Scope

When a project “is late” it means that (for whatever reason) what was expected to be ready at a given time, isn’t. Date doesn't match scope - or vice versa.

Now, let’s assume that I fix the date, i.e. I commit to delivering my software on a certain date, but I don’t commit to an exact scope by that date. Instead, I make sure that the scope is allowed to flex so that I can deliver only whatever is ready by the time I reach the date that I have committed to. As you all know by now this is what Agile is all about. It is OK to commit to either date or scope, but never both.

Is it now possible for my Agile project to “be late”?

And I believe the answer is "No". An Agile project can never “be late”, because it is impossible. “Late” is a concept that does not apply to Agile projects. By allowing the scope to flex you effectively remove one of the two required factors for becoming late in the first place. Consequently; if the project is late, then it is not an Agile project.

One could of course argue that “being late” can also be when a stakeholder desires the scope to be in a certain state sooner than it actually is. However, in my book this is called “wishful thinking” on the behalf of the stakeholder, and it has nothing to do with the project “being late”. That needs to be handled by proper management of expectations – which Agile and Scrum projects in particular solve by opening up the project with full transparency to stakeholders (velocity measurements, release plan based on velocity, highly visible burndown charts, deep involvement of customers and other stakeholders, short feedback loops with regular deliveries and demos, etc).

What the stakeholder above has to realize is that the scope will be delivered as fast as reality allows. The state the project will be in on any given date is determined largely by the “context” in which the project is being realized. When I say “context” I mean all those factors that are affecting how fast the realization can happen: the competence of team, the cooperation of team members, the availability and effectiveness of tools used, the ability for the team to focus on the task (i.e. disturbances), the mood & motivation of individual team members, the availability of information, the clarity/understanding of what is requested, and so on.

So, let’s agree to never again claim that an Agile project is “late”.

OK? :-)

Scrum. You're doing it wrong when...

2009-11-05T08:16:00.007+01:00

…your sprint is so long that you feel you have to make a time-plan for it.
…your Scrum Master tells people in the team what to do and when.
…the team report their progress to the Scrum Master.
…estimates are not done collectively by the team-members who will be doing the work.
...you hear one team-member say to another; “Sorry, I’m really focusing on my own tasks here in order to finish by the end of the sprint, so you will have to solve your own problem yourself.”.
…all stories in the sprint finish on the last day of the sprint.
…lower-priority stories in the sprint gets done before higher-priority ones.
…the team has as many stories in progress as it has team-members.
…the Scrum Product Owner can’t explain every single story in both the Product Backlog and the Sprint Backlog.
…you still think it’s a good idea to write down all requirements before you start coding.
…you consider customer feedback a risk.
…the team doesn’t know who their Scrum Master is.
…your system testing happens all at once, after a couple of sprints.
…you still think that a detailed Gantt chart is a useful planning tool for the software development tasks.
…your sprints are 6 weeks long.
…the team’s burn-down has been flat-lined for three days and they’re not discussing why.
…you think the customer knows what he wants.
…you think the team knows how to build what the customer says he wants.
…the team doesn’t know how much work is remaining of the sprint.
…you think it’s OK if the team has 45-minute Daily Scrum meetings.
…you come running to the team after a sprint and yell “Hey, we have to release now!” and someone in the team responds “…Oh, but wait, we still haven’t done any system testing… And there’s some more code review to be done too.”.
…the team doesn’t know when and where the sprint demo will be held.
…the team constantly over-commits.
…you think it’s OK that the team has their Daily Scrum meetings every other day instead; “It’s their choice, and meetings take so long, so….”.
…the team is persuaded to add more to the sprint than they really want.
…the team accepts to include stories that have not been estimated.
…you don’t collect feedback from the customer after every sprint.
…the team doesn’t know who their Scrum Product Owner is.
…your team is larger than 8 team members.
…the team doesn’t know what the sprint goal(s) are.
…your Scrum Product Owner misses a sprint demo or a sprint planning every now and then.
…the team decides (or doesn’t follow or care about) the order of the stories in the sprint backlog.
…you think the Scrum Master is sort of like a Project Manager.
… you think Scrum doesn’t require or need planning.
… you think you can’t have a deadline or commit to a scope "because you’re using Scrum".
… you think you don’t need to document "because you're using Scrum".
… you commit both to a certain scope and to a certain date.
...you need to add stuff to the sprint half-way through.
…you think the Scrum Product Owner is sort of like a Project Manager.
…the team doesn’t know their velocity, or the meaning of it.
…the team haven’t met the Scrum Product Owner for a week.
…the contents of a sprint demo comes as a surprise to the Scrum Product Owner.
…the team doesn’t know or understand the overall vision of what they’re working towards.
…you can’t release the software after every sprint.
...Scrum doesn't work.

Follow-up on the Scrum Practitioners meeting Oct 15th

2009-10-27T09:07:00.005+01:00

Thank you everyone who participated on the Scrum Practitioners meeting on the 15th of October at Axis Communications in Lund.

Some of the things discussed were;

Distributed Scrum; does it work? A few of the participants had some experience in this, and the opinions were mixed. Bottom line is; yes it is certainly possible to get it to work. I.e. it doesnt automatically fail. It requires work, though. But that is true for any methodology; if you have team members on a distance you have to deal with the overhead cost in terms of extra communication, tools, activities, travel, whatever. Depending on how the distribution looks you can try different approaches; divide into teams per site, or distribute teams and work with conference solutions, digital whiteboards, etc for the daily meetings. Experiment. Distributed teams and distributed Scrum is not ideal, but if you're faced with a situation where you have a distributed team, don't automatically rule out using Scrum.
How/if sort out design-issues before sprint start? The problem discussed was that of things taking a long time to finish if you do "design stories" in one sprint to discover what/how to implement, and actually implement in in the following sprint. With 4 week sprints it can take you 8 weeks to finish a story, even if it is not that big. The concept of "spikes" was discussed, i.e. prototyping and delivering a piece of working and valuable code in each sprint, rathen than doing "only design" separated from "only development". If you do "design stories" it may be wise to timebox rather than trying to estimate it, because the definition of done for "design" is hard to grasp. Another tip can be to consider shortening the sprints, to e.g. 2-3 weeks instead of 4.
The problem of traditional, conservative customers, demaning fixed-price-fixed-scope contracts was discussed. The concept of "Money for Nothing and Change for Free" was discussed as an interesting theory. None of the participants had experience with that approach, though.
Coping with less-than-fulltime resources in a Scrum Team: The problem of traditional customers and product owners who do not realize that Scrum requires a much higher degree of involvement. Not sure we came to any intelligent conclusions on how to address this problem, other than the importance of realizing that it is a problem if the customer/PO is not involved.
Problems with large team size was discussed, and consequently the problem of daily scrums taking too long time.
Is the traditional project manager automatically the best choice for Scrum Master when introducing Scrum? Not necessarily, was the conclusion. This seems to be the default choice in the industry, but it is not necessarily the best. It depends on your organization, the people, the problem domain, etc. The Scrum Master may be a good choice for the Project Manager, but Scrum Product Owner may be too.

Several other interesting things were discussed too, but I will not go into more details here.

I'm looking forward to the next meeting.

October meeting Scrum Practitioners South of Sweden

2009-10-05T09:11:00.005+02:00

I'm planning the October meeting for the network Scrum Practitioners South of Sweden. I'll update this blog post as details are decided. For now, the meeting is planned for Thursday October 15. It will be between 17:00 and 19:00, hosted by myself, in Axis Communications' office at Emdalavägen 14 in Lund.

The topics of discussion are (under construction ;-)):

* Distributed Scrum; Does it work?
* How/if sort out design-issues before sprint start?
* How do I get the product owner more involved?
* Coping with less-than-fulltime resources in a Scrum Team
* Is the Project Manager the best choice for Scrum Master when moving from a Traditional to a Scrum organization? Who decides, and who is the best choice?
* Is the "Product Manager" the best choice for "Scrum Product Owner"?
* Story formulation, estimation, story breakdown into tasks and prioritization
* ...<insert topic here>

Follow-up on the Scrum Practitioners meeting Sept 7th

2009-09-16T08:05:00.002+02:00

To follow up on our first meeting in the Scrum Practitioners South of Sweden network;

I don't intend to recap all the discussions and conclusions on the meeting, but rather I just wanted to briefly list the major topics discussed as I recall.

The meeting was held as planned at Labs2 in Lund with a total of 8 participants including Mathias Thinsz (Product Manager & Scrum Product Owner at Labs2) who was our host.

Mathias went through and explained the major parts of Labs2's Scrum implementation. It seems to me that they have a very solid and healthy view on how they development software. It's a clean and well-thought-through implementation of Scrum. A great example on how well Scrum works.

Related topics discussed were;
* How Labs2 use JIRA and a custom developed web application for managing the backlog
* How and when Labs2 estimate their work in the backlog, and how they use the storypoints unit
* How Labs2 has divided their development organization
* How and why Labs2 have divided their development time into fixed-length iterations
* How Mathias (as Scrum Product Owner) prepares for a new sprint and who is involved when
* How and why Labs2 use "Sprint Goals" as a major tool for directing development inside a sprint

There were tons of other interesting questions and discussions as well, but too much for me to go into here.

The general feeling, in my understanding, was that the meeting was interesting and inspiring and everyone had a lot of questions and topics which they wanted to continue discussing.
The number of participants was optimal and the length of the meeting as well. Hence; my conclusion is that there is a definate interest for meeting again and that the next meeting should be in the same format.

I want to take the opportunity to thank everyone who joined and not least Mathias Thinsz and Labs2 for the openess with how they work!

Mon Sep 09: Meeting Scrum Practitioners South of Sweden

2009-09-04T08:34:00.004+02:00

On Monday the 7th of September 17:00-19:00 we're a bunch of people meeting at Labs2's office in Lund (Sweden :)) to discuss how Labs2 are working with Scrum. Anyone intersted in joining and sharing experiences is welcome.

Join the LinkedIn group "Scrum Practitioners South of Sweden" and post your interest if you want to join the meeting.

My idea with this is to get software professionals in the Öresund region to meet informally, network, and not least share experiences and help eachother improve our way of working with agile software development. Meetings will be hosted by a different group member each time, and the discussion theme will vary from meeting to meeting.

See you there!

Scrum but... test

2009-09-03T11:34:00.003+02:00

This has been around for some time, but if you haven't taken it already then do it. It's a good way to compare yourself to others and help emphasize your points of improvement.

Go to the Scrumbut test!

I just took the test and my team currently scores 7.1 out of 9.

Estimating & prioritizing the product backlog

2009-07-20T08:15:00.001+02:00

One of the things that I’ve been struggling with getting my head around since I started using Scrum is; how much effort do I put into estimations of the product backlog, and do I or don’t I estimate all at once? And do I re-estimate stories as I go along and learn more?

Originally I’ve been under the impression that I shouldn’t attempt to estimate it all up front. It's easy to argue that it is rather un-agile to do so: because I would be spending effort on low priority stuff, which is a big risk for waste. On the other hand, if I don’t estimate the backlog up front, I can’t tell its entire size and consequently I can’t (potentially) tell what I will have ready by when.

So far, I’ve been trying a couple of different approaches. One of my favorites has been to estimate in priority order up to a few months worth of stories (a "handful" of sprints). The remaining unestimated stories I apply an average size to based on the estimated ones. I know, it’s risky, but it’s at least something. And obviously, as time passes, I estimate more and more; trying to keep about 2-3 months worth of stories estimated at all times. For some time this has been my approach to release planning without estimating everything up front.

A problem I only recently realized I’ve had is that this approach requires me to prioritize before estimating; and consequently my prioritizations can’t take size into consideration. My prioritization has only been made based on “value” (whatever that is) without regard to cost. Is that optimal? I think the answer is No. Given I have a choice prioritization should take size into account. Or at least, if I don’t take it into account it needs to be a conscious decision not to. Which brings me back to my original question; how do I do it, do I estimate everything up front or not?

Recently, I’ve decided to try changing my approach (inspect and adapt, right?) and actually estimate the entire backlog up front, before prioritization. The good thing is that estimating in story points is very quick (the team usually doesn’t have to spend more than a few minutes per item) so the time spent on (potentially) low-priority items is minimal.

Once I have the story point estimates of all backlog items I use a prioritization technique called “Theme Scoring”. Let me try to describe it with a simple example;

The first step is to decide on a couple of aspects ("selection criteria") to judge stories on; for example “Improves user interface”, “Simplifies maintenance”, “Brings revenue in Q3” and so on. The combined criteria will represent what you call “business value” so whatever you put in that term should in one way or another be reflected by those criteria that you pick. Try, however, to keep it rather simple e.g. up to a maximum of 4-6 criteria or otherwise your scoring will become very tedious and the effort you spend on scoring (and consequently the risk of waste) I think will be too great compared to the benefit.

Once you’ve decided on your criteria you need to weigh them relative to each other. Maybe “Brings revenue in Q3” is the most important criteria to you right now, so let’s give it the weight of (for example) 5. “Simplifies maintenance” is not that important right now so you give it the weight 2 (it’s less than half as important as the revenue criteria). “Improves user interface” is almost as important as the revenue aspect, so you assign it the weight 4. Here are your criteria and their weights so far; “Brings revenue in Q3” = 5, “Improves user interface” = 4 and “Simplifies maintenance” = 2.

The next step is to go through all of your backlog items, one by one, and score them on a scale from 1-5 in terms of how they affect each of your selected criteria. For each criteria you can usually identify a “baseline story”; a story that gives you a fair amount of value in terms of that criteria, yielding in a 3 point score on that 1-5 scale. There should be stories that yield in a higher score as well as lower. Stories should be compared to that baseline story for each criteria to determine if it should get a higher or a lower score than the baseline.

This could be your result after going over all criterias for all stories;

The numbers marked red are the baseline stories for each criteria. The black numbers in the criteria columns are grades I assign each story on a scale from 1-5 relative to the baseline. 1 and 2 means much lower and lower than the baseline, and 4 and 5 means higher and much higher than the baseline. A grade of 0 means that the criteria doesn’t apply to the story. The Score is simply your grade multiplied by the weight, for each criteria.

Given your selected criteria and their weights, in the example here Story 3 is the one giving you the most value (37), followed by Story 2 (18) and then Story 1 (17), Story 5 (8) and Story 4 (6).

Now, it is time to compare value to cost (estimated size in Story Points);

So far this technique has been possible for me to do all along, even while I have been lacking estimates of parts of the backlog. But the following part – taking cost into the equation – obviously requires me to have the estimates. Putting the Benefit in perspective to Cost is a great idea; it gives me the opportunity to maximize benefit while minimizing cost.

So, divide Value (your total score for each story) with Cost (measured in story points) and sort your backlog by the result: the higher the value the more value I get back on the effort put in:

The above table is the resulting suggested prioritization based on the Theme Scoring technique and on calculating value/cost. Obviously it is worth noting that this is a suggestion only. It is an aid that I can use when prioritizing my backlog, it’s not something that replaces my thought-process – I still need to think and consider every item and e.g. be careful that I don’t miss any dependencies between stories (yes I know, we shouldn’t have any dependencies between stories, but sometimes we do anyway! :-)).

As always, I’m interested in input on how other people do things…!

Technical debt in Scrum projects

2009-07-08T08:05:00.007+02:00

Code deteriorates with age. The older the system, the worse the maintenance. Why is this?

One answer may be that it is because of the maintenance itself: the things done over the years to keep the system afloat and the new features added, removed and changed. And people come and go. Some quit the team and new ones join. Sum it all up: deterioration.

Let’s talk about some of the causes and effects of deterioration.

Reinventing the wheel
An example of this is when a system contains several versions of the same function implemented at different times, often by different people, with only small differences. A similar situation is where a design (architecture) problem is solved in different ways in different places in the system. Either way, reinventing the wheel increases the complexity of the system and makes it harder to maintain. Developers won’t know which function is used where or even if they are still used at all (and consequently leave them there).

This is a negative spiral; the more complex the system gets the harder it is to understand, and consequently the harder it is to know if “my” problem has been solved already somewhere – which might lead me to unintentionally write my own version of a function that actually already exists. The lesson is to not reinvent the wheel. But it is easier said than done.

Maintainability is not something you can just add down the road. Building a maintainable codebase starts from the first line of code and it never ends.

Fear of changing what works – Legacy code
Once a system reaches a certain level of deterioration, people will be afraid to change certain parts. It’s often some critical and central component. Since it is central and critical it means it has been patched up numerous times, and changed and adjusted and changed again – over a long time. And hey, it works – now. And since it works, now, and since the code is so complicated and confusing, the developers don’t dare touch it. Consequently any features that would result in work in that critical component will be held back.

The way I see it, one of the root causes for this, is lack of regression testing abilities. With proper means for continuous and whole-covering regression testing, there is little to fear even when it comes to modifying old “legacy code”. Otherwise there are only two options when it comes to modifying legacy code; a complete rewrite, or just not touching it at all.

Lack of (efficient) regression testing abilities
The purpose of regression testing is to verify that every part of the system – even the less obvious parts – still works after a change or addition somewhere has been completed. The point with regression testing is to test a lot, over and over and over again. If you are in a situation where regression testing requires immense resources (which it does if you do it completely manually), then you will not be able or willing to do it as often or as extensive as is required. As a consequence your regression testing become inefficient, or even non-existent.

One of the steps in the right direction is to start automating your tests, and start running those automatic test cases continuously. The challenge, of course, is to;
(A) find a practical means - a tool - for automatic testing of your type of code, and
(B) figure out where to start.

I can’t help you with A (...psssst: JUnit, CPPUnit, CUnit, PHPUnit), but for B I suggest you just start somewhere. Don’t try to do it all at once – you won’t make it! Instead, just pick a simple starting-point, for example the most recent and newly added feature/module/component/part. Forget about the old stuff for now, just add automatic testing for the new things from now on. Something is better than nothing.

People joining and leaving
This is, unfortunately, unavoidable in most projects that are longer than just a few months. It happens. Either people get reassigned, or they choose to quit. And best-case, people join the team. Apart from a change in productivity caused by a team member leaving or joining, it obviously has an effect on the code itself too. People leaving will take knowledge with them, and people joining bring in new ideas (and misunderstand parts of what exists, too). One way to minimize the effects of this is to organize into “Feature Teams” that has a lot of close cooperation and joint commitments (like – tada! – Scrum suggests). This way you naturally spread knowledge among several people. It is also a pretty effective way of introducing new team members into the groove of things.

The classic method to minimize the problem of people leaving and joining is to write documents. I however argue that this is not the silver bullet for spreading & retaining knowledge. In fact, I think it's dangerous to rely on documents as the main tool for this; documentation is an extremely cost-inefficient and overrated way of transferring knowledge, and something that is often forgotten is the cost of keeping documentation up-to-date. As soon as documentation falls behind it becomes untrustworthy – and untrustworthy documentation will cause confusion and misunderstandings, and in the end no one will dare rely on documentation, and the time writing and updating it up to that point becomes waste.

In my opinion the guideline should be: don’t over-document – document just enough. Code Comments, I think, is a great benchmark of what level of documentation is “enough” for most situations. And remember; one excellent thing about code comments is that it is automatically (well…) kept up to date as the code changes – there’s little or no added cost for keeping it up to date.

Oh, and for the record, I’m not saying “Don’t write documents!”. If you really need to document then of course you should. I’m merely suggesting that you at the very least question the reason for doing that effort, and that you don’t forget to take into account the cost of keeping the document up-to-date as the system evolves.

Taking shortcuts - the Dirty that remains long after the Quick has been forgotten
“Well, we’ll do the quick-n-dirty fix now, just to get it done, and then we’ll go back later and clean it up...”. Have you ever said or heard something along those lines?

Short term gains such as reaching some immediate deadline makes it tempting sometimes to take shortcuts, and often it’s, sadly, a conscious decision. The problem with shortcuts is that they seldom or never get fixed afterwards, because there’s always that next deadline coming up with a new bunch of stuff to do with a new bunch of shortcuts that “has” to be made.

Doing things right from the beginning often requires a little more effort up front. And I think that different times call for different ways of acting. Sometimes it might be a correct decision to look only at short-term gains and cutting down on the immediate effort, and forgetting about those long-term consequences and drawbacks. But many teams and managers, I think, tend to be shortsighted by default – even when they don’t have to and there would in fact be room to do things properly. And in that case it is a matter of attitude (and competence). Do you do things fast and sloppy now and accept to pay for it later, or do you let it take a little longer now and reap the benefits of it later (for example in terms of costs saved on maintenance)?

It’s a challenge to figure out what is a shortcut and what is not. Remember Lean Software Development and the idea of “Extra effort” (and “Extra features”) being Waste. How do you know what is “Just enough effort”? There is no default answer to that. It depends. It’s up to you to figure that out for your system and for your business. But by figuring that out (or deciding on it) you will know what level to strive for; and anything below it is a shortcut and should not be accepted.

Don’t forget that you need to make sure that whatever your level is, it should be gut-felt by every team-member.

Bug fixes
Bug fixes have a tendency to deteriorate code. Enough patches in one place and the code will become more and more messy – at least if you have other problems too that cause code deterioration, such as people coming and leaving, inability to do regression testing, etc.

Bugs found in a production environment are often time critical to remedy. It can be tempting to take shortcuts to just fix the problem quickly and get a patch out there. But if you do that enough times but never take time to clean out the mess, you are destroying your system. See section above about taking shortcuts…

Summing things up – dealing with Technical Debt
A nice way to think of this deterioration of code is to think of it as “Technical Debt” - a term coined by Ward Cunningham.

Technical Debt is a long-term loan that we for some reason choose to, or have to, take from ourselves in order to achieve some short term gain. The Technical Debt increases for every individual loan and the debt never just goes away by itself. We have to pay “Interest” in the shape of things taking longer to complete because of this deteriorated code, and the only way we can decrease the Interest is by decreasing the loan – by paying “mortgages” e.g. by refactoring.

The most common (and worst) approach to dealing with Technical Debt is to ignore it. To pretend it doesn’t exist, and just push development forward without considering whether or not we’re taking loans.
The better approach to dealing with Technical Debt is to recognize it and have an active plan on how to deal with it in various situations.

I suggest you deal with technical debt into two steps: First stop increasing the debt further, and secondly start decreasing it. Only once you know that your debt is not increasing, you can start actively working with decreasing it.

To support you in the first step I suggest you insert a row in your Default Definition of Done that says “The Technical Debt has not increased”. It sounds trivial, but the intended effect is that it recognizes that it is OK that things take longer to complete if done in a way that doesn’t increase the debt - e.g. that it is OK to not take shortcuts. Remember to constantly remind people about this, and really do put your the money where your mouth is. Whenever faced with a choice, make sure you consider that the decision should be in-line with the attitude of letting things take longer in order to not increase the technical debt.

Once you’ve gotten used to that approach (it probably takes you a while and, if nothing else, will probably cause your velocity to drop significantly at first), the next step is to change the Definition of Done to instead say “The Technical Debt has decreased”. This is intended to recognize the fact that it is OK to also do some refactoring of things surrounding the current implementation “while you’re at it”. For example, urge all developers to modify the methods above and below the one currently worked in - even though it wouldn’t be necessary to complete the story itself! This way, for every new story completed the existing debt will decrease.
This type of refactoring will puts a lot of demands on your regression testing abilities. If you don't already have an automatic testing environment I suggest you start with introducing that first. Refactoring working code will (as explained in a section above) will otherwise be much too scary.

That's it from me for now. As always I'm interested in hearing other people's experiences and opinions in this matter.

Scrum Practitioners South of Sweden on LinkedIn

2009-07-03T08:00:00.002+02:00

I just started a LinkedIn group: Scrum Practitioners South of Sweden.

The idea is to collect a bunch of people in the south of Sweden who want to meet face to face regularly, in an informal manner and share experience of Scrum, Agile and related topics with the intent of helping each other out and improving how we work.

Not sure if such a network already exists but I certainly see a need for it personally.

Follow this link if you're interested in joining.

Done Environment

2009-06-26T12:43:00.002+02:00

In addition to intelligent version control, I like to have a set of “test environments” that map closely to my Agile Version Control policy.

First, it is necessary to have some sort of “Development Environment” where the most recent code from the Development Branch in the version control system can be tested and worked on.

Similarly, I like to have a “Done Environment” (sometimes also known as “Next Release Environment”, or “NR Environment”) where the most recent code from the Done Branch can be seen. The power of this environment is that it allows (forces) you to integration test early. If your Done Environment is an accurate representation (e.g. replica) of your live environment then you will be able to integration- and system test as early as the first story being delivered there – and I always include in the Default Definition of Done that the story should be working and available for demo in the Done Environment. In fact, I like to setup an automatic build and distribution script here, that every night packages the most recent version of the Done Branch in the version control system and puts it on the Done Environment. This way you know that you will always be able to see the stories that are Done-Done on in this environment, without any risk of them being infected by code in progress. This also means that once the release day comes you have (hopefully) already done a lot of the steps required, and consequently making a release becomes less of a deal than it otherwise often is; avoiding or at least minimizing the Big Bang problem.

If you’re short on servers, server-space and/or money (actually, regardless) I think it is a great idea to use virtual servers for hosting the environments. It gives a great flexibility when it comes to setting up new environments and cloning existing ones. If you have a live environment consisting of 40 physical servers, and you want to replicate that environment as a Done Environment, it’s not always practical to buy 40 new physical servers. Especially not if you have three teams and each has its own Done Branch, and consequently needs its own Done Environment (3 x 40 servers ;-)). But maybe it is OK to buy just a few physical servers, and instead run the Done Environment servers virtually. Virtual servers is a great and powerful tool – try it. You can find a list of software here: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines

The velocity log - another Scrum tool

2009-06-22T07:20:00.005+02:00

Do you measure velocity? If so, how do you record it? (And if not: start!)

You measure (or should be measuring) velocity in order to be able to adjust the intake of future sprints, allowing the team to optimize the size of its commitment, facilitating for keeping a constant pace. What you should be measuring, at the very least, is the number of story points completed according to the Definition of Done in each sprint. This is, as you probably know, called Actual Velocity. You should also measure how many story points the team originally thought they would complete when they started the sprint (on the sprint planning meeting). This is called the Estimated Velocity.

I like to do the counting as part of the Retrospection. The team review what they have delivered and we do the math and record it.

The actual (and estimated) velocity are dependent on the context in which the team work; plenty of disturbances, impediments, new people etc will yield in a lower actual velocity. The velocity is also dependent on the sprint duration and on the team size. You want to keep both as constant as possible, so that the team really has a continuous pace – a “beat”. However, there’s a reality, where people have vacations and there are parental leave and educations, and there are holidays and day-before-holidays and what not. Despite best efforts, the sprint duration and the team size will vary over time – a lot, sometimes. This makes it hard to compare the velocities of different sprints; one sprint resulted in 25 SP and another 40. Was that because the environment was such that it allowed the team to complete almost double the amount, or was it because the sprint had a few extra days and a couple of people less on vacation?

I like to remove sprint duration and team size from the equation so I get something comparable over time (across sprints); because variances in velocity should point out differences in the context, and not differences in the sprint duration or team size.

For this reason, I record also sprint duration (in number of working days) and team size (in number of people), together with every pair of velocity (actual + estimated). I try to keep the numbers to integers, but sometimes I have to count “half” people (because of e.g. parental part-time leave, educations, or similar). I try not to go into more details than halfs – if I’m just consistent in the way I measure and in the level of detail, it won’t matter in the end – so why waste time and effort trying to figure out if I have 5.18 persons or 5. Anyway, with this information (Sprint Duration and Team Size) I can calculate the number of available man-days in the sprint (number of days x number of people), and then I can use that to calculate the “Estimated Story Points per Man-day” and “Actual Story Points per Man-day”. It will say something like “1.5”. It means that for every man-day the team completes 1.5 Story Points.

I’ve made an Excel (I always do ;-)) for recording velocity like this. I call it “Velocity Log”. It also has some nifty charts to plot velocity and trends over time. It’s available for download here:
VelocityLog-v1.xls

As an example, here’s the velocity of my current team over a couple of sprints:

Let me know your thoughts.

Why estimate using Story points?

2009-06-17T08:45:00.002+02:00

The other day I was talking about why bothering using Story Points instead of just estimating everything in hours. Here's a link to the slides.

Traditionally, the aim has been to try to estimate time by looking intensely at a problem (or a task, activity, requirement,…) – so traditionally there’s a lot of effort spent on breakdown, analysis & estimation. But at the same time, we all know that estimating time is very hard and often pretty inaccurate – because software development is research-oriented and there are things that surface only once we start digging in, which then change the picture. We all know this – yet those time estimates tend to become commitments down the line.

I argue that it’s not only the problem (task, activity, requirement,…) that is hard to foretell; the environment where we solve the problem also has a significant effect on the amount of time it will take to complete. And often the environment is much harder to foretell than the problem itself. So, in the end it doesn’t matter how long time you spend analyzing, you will still have a large unknown in the shape of the environment.

The point I want to make is that estimating in Story Points is about recognizing that Time is a result of the Size of a problem solved within a certain Context.
Time = Size x Context

Story Points is a unit for representing Size. This way Context becomes something we can grasp by comparing Size to Time – which is why we measure Velocity.

Agile Version Control - another magic bullet

2009-06-08T22:32:00.010+02:00

I have previously posted articles about what the magic bullet of agile software development is; to achieve flexible scope. As you may recall, the article talks about the three most important aspects of your Scrum implementation which make up the fundament for being able to achieve this scope flexibility:

Strict prioritization (having it and following it)
Definition of Done (having one and following it)
Story breakdown & formulation skills (understanding its importance, and constantly trying to improve)

In hindsight, I feel I have left out one important success factor when it comes to reaching a truly flexible scope; version control.

Most of us use one form of version control or another, but in my experience it is often (mistakenly so) only a matter of choosing the tool. The options are many, and range from “open source” to “incredibly expensive”. Once you decide on the tool, you use it for version tracking your codebase by checking out, committing and merging files. Perhaps you use branches in order to separate some work from other. But you may be missing an important tool; an Agile Version Control Policy.

If you do Scrum or some other form of Agile Development, you want to be able to actually finish your product increment every iteration, right? The purpose is to be able to achieve a flexible scope so that you can potentially ship on any day, at any given time. And what I am trying to get to here is that it doesn’t matter how well you succeed with your adherence to a Definition of Done if you don’t combine it with applying an Agile Version Control policy (completely regardless of which version control system you’re using). If you don’t have an Agile Version Control policy in place you risk mixing up your “Done-Done” code (code and assets related to User Stories that have been completely done according to the Definition of Done) with code which is in progress.

And guess what: if you mix Done code up with Code in progress, you’re no longer able to release at any given time!

Without Agile Version Control, shipping your software will mean that you first have to perform a series of tasks related to isolating code that is Done-Done from code that is in progress at that moment, and also often tasks related to doing last-minute polishing “while you’re at it”. The result is that shipping code is hard and a “big deal” and something that takes a lot of time. And what else does it cause? Yeah: crunch.

So how do you achieve this separation of Done-Done code from code in progress? Well, there are plenty of good articles out there to read: this one for example, written by Henrik Kniberg.

In short, what you need to do is dedicate a branch for containing that Done-Done code (let’s call this branch “Done Branch”), and define some rules for it. For example "all code on the Done Branch must adhere to the Definition of Done", "no code may be published to Done Branch without being reviewed", "code may only be published there by the branch owner", and so on.

Simplicity ftw! Here is a very simple Agile Version Control policy that I’ve been using;

Live Branch
Description: Create a branch from the trunk when you make a public/final release, that you will maintain for a while (until next release). Maintenance (patching) happens (for sake of simplicity) directly against this branch.

Branch policy:

All code in Live Branch must be tested and working.
All code in Live Branch must compile & build. In short: all code in Live Branch must follow Definition of Done.
Code may only be published to Live Branch by branch owner (who may delegate it).
Patches on Live Branch are immediately published to Trunk as the last step, and any conflicts are resolved on Trunk until code adheres to the branch policy. Note that you don’t resolve the conflicts on the Live Branch because it would imply updating Live with whatever is on Trunk (which can be a lot of changes).

Done Branch (which happens to be the Trunk, aka Main)
Description: Use the trunk (aka main branch) as “Done branch”. This will contain only code from stories that are Done-Done.

Branch policy:

All code in Done Branch must be tested and working.
All code in Done Branch must compile & build. In short: all code in Done Branch must follow Definition of Done.
Code may only be published to Done Branch by branch owner (who may delegate it).

Development Branch
Description: Create a branch from the Trunk in which the team works during the sprints. The branch exists across several sprints.

Branch policy:

All code in Development Branch must compile & build.
The Development Branch should be updated from Trunk often (suggest daily).
Conflicts with Trunk are resolved in Development Branch until code adheres to the branch policy.

Remember to update your code often from Development Branch (i.e. spark merges as often as possible!).

This simple policy relies only on three branch types; Live, Done and Development. Your Trunk (aka Main) is your Done Branch. There is a drawback with this policy though; if you have multiple teams who are developing (and finishing) stories which are supposed to be able to release independent of each other, then you will not be able to use the policy above (because Done-Done code from the two teams will be mixed up in the Done Branch). But, do you really have the requirement of teams to be able to release code independent of each other at different dates? Think carefully about this – do you really? If you do, fine: in that case you have to have a slightly more sophisticated setup (with a separate Done Branch + a separate Development Branch per team, and consequently you can’t use the Trunk as your Done Branch). You can read more about such a setup (Agile Version Control in a multi-team environment) in the paper by Kniberg that I referenced above.

As always, I’d be glad to hear about other peoples’ experience in this area. Please leave comments.

When Scrum meets traditional quality assurance

2009-05-28T16:15:00.003+02:00

The contents of this post has been growing on me for some time. It’s about the challenge of working with Scrum in an organization where there’s a lot of traditional (eg waterfall based) ways of doing things.

Let’s say you’re the manager of a team that develops and maintains some software system. Let’s say you’ve successfully introduced Scrum in that development team and they’re doing well and things are progressing. But, you don’t have any “Testers” on your team. Yet. Whatever testing is done is performed by the programmers themselves, in a more or less ad hoc way. Now, there’s a whole “QA Department” in your company, with resource(s) available for you. But it is an entirely different department that is not under your control. All you can do is request and allocate one or more resources from QA – but you’re not in a position to affect how they work. They are used to working according to a waterfall style project model. The processes that they follow rely on a “requirement specification” to exists ahead of time, before the implementation begins, together with a detailed and approved UI design ahead of time, a detailed and approved system design ahead of time, and so on. So the Testers from the QA Department will not be able to assist with any testing unless you have these things in place for them to start building test cases and test specifications on. What do you do?

One of the most central challenges for me right now with regard to Scrum & Agile Development is exactly that: How to continue working with Scrum in my development team and still manage to cooperate and integrate smoothly with another department that is not yet working according to Scrum or the Agile philosophy.

Is it possible to follow agile principles in only one part of the software development process?

So far, since I haven’t had any testers on my team, I have more or less been ignoring the problem and simply done things cowboy-style. I’ve run the project as a pure Scrum project, with user stories in a Product Backlog, with a Scrum Product Owner, with velocity measurement, etc, etc. And – in line with Agile – we’ve worked in a strictly prioritized order, so consequently we haven’t spent much time yet on those lower priority backlog items, and they would certainly not yet be possible to use for formulating any “Test Cases” in the QA Department’s meaning of the word. But the time is coming where my Scrum team and their way of working will meet the QA Department and its way of working. Will the two fit?

Don’t misunderstand me! I’m really looking forward to getting those QA resources on the team, and I’m 100% confident, regardless of philosophies and project models, that quality will benefit from the addition of QA resources – no question about it. But I’m worried that the benefit may be at the expense of “quality” in the development process itself (in the sense of me having to “sacrifice” certain Agile cornerstones).

In my view, “testing” and “quality assurance” is not something that is applied on top of programming as a separate stage. Quality assurance is something that is intertwined with the entire software lifecycle; from idea, through analysis, design and development, to integration and system testing, rollout, handover and maintenance. And since it is all intertwined inside the software lifecycle, it is not possible to limit Agile philosophies to programming only. To be precise, and to answer my own question above; No I don’t think you can be “agile” in an environment where you finalize a “Requirement Specification” at the beginning of the project, and then base everything on it; plans, designs, test cases, etc.

So, my plan is that together with the coming QA resource we will try to find a way for us to meet half-way; to find a way of working that requires minimal change in the QA processes and still minimal change in the Scrum implementation. I really think this can be done.

Here’s the approach so far;

What does the QA Department do? Well, they test software. What information do they need in order to be able to test? Well, they need a test plan, and they need a set of test cases. And what information is needed in order to create the test plan? A schedule, probably. Great! I can deliver a schedule, no problem. I just have a slightly different way of creating it than traditional projects.
And what information is needed in order to formulate test cases? “Requirements”! Could I deliver the backlog as “the requirements”? I suspect no. Darn. Why? Well, the User Stories alone are probably not detailed enough for QA to base test cases on; the level will be too high and there will be too much room for ambiguity, especially before we start working on the story. However, once we’re done with each story we’ll know more about the requirements. I don’t want to have to deliver the requirements ahead of time.

But wait. Didn’t I say that QA is an integral part of the software lifecycle? Yes I did :-). So what is Scrum? Iterative and Incremental. Can’t the “Requirement Specification” also be created Iteratively and Incrementally? And consequently; can’t the set of Test Cases (the Test Specification) also be created Iteratively and Incrementally?

So, the goal for us now is to make incremental deliveries of the “Requirement Specification” and of the “Test Specification”, in parallel to the incremental deliveries of software – sprint by sprint. We’re just getting started and are working out the practical bits in order to achieve this. For example, we’ll include in our Default Definition of Done that Requirements should be formulated. Secondly, we’ll keep track of which Stories are Done-Done (meet Definition of Done) at the end of each Sprint. The tester will use that information as input for his work. So at the end of each sprint there’s a new set of finished Stories accompanied with Requirements, that the tester can use as input to create a Test Specification increment. This way, the formulation of test cases is done one sprint behind the development. It might not be optimal, but considering that the tester is not full-time allocated to the development team, doesn’t sit with us, and that I’m not really attempting to reshape the entire QA department’s process all at once at this point :-), I think it’s a fair construction and a good trade-off.

So. That’s where I am right now. I’ll post an update as we progress!

I’d be glad to hear about other people’s experience about this type of situation.

How to adopt Continuous Refactoring?

2009-04-22T17:39:00.006+02:00

This is a question that I have been asking myself, as well as received from others, a bunch of times. I am posting it here in the blog since I don’t have a ready-made answer. Please leave some comments on this if you have suggestions.

To clarify the question, let me begin by setting the scene for you;

Let’s say you are introducing Scrum. The team is, as always, a bunch of developers with varying experience, competence, and ages. You have read my blog ;-) as well as other even better sources of Scrum & Agile ideas and opinions, and you understand that the concept of “continuous refactoring” is central. You realize that anything is less likely to succeed if you try to solve everything at once in one single shoot; so it is better to work iteratively and deliver in increments, in order of priority. How do you get this message through to the developers, so that they really adopt it, wholeheartedly, in practice?

Because the problem here, in my experience, is that most people take a lot of pride in their work and “continuous refactoring” is easy to mistake for “rushing things out”. Some people just can’t get used to the idea of not attempting to do it “perfect” or “complete” from the beginning, and thereby find it extremely hard to deliver something that “just works” and is “good enough” for that particular product increment. Obviously, people are different (and so are programmers ;-)), so some will easily adopt this way of thinking, while it will be much harder for others.

How do you do it? It probably has much more to do with psychology and leadership than with Scrum per se, but I still think it is a relevant topic to bring up within the boundaries of this blog.

My suggestion on how to go about this is to educate. Talk about it. Have trainings. I have had lengthy discussions about this topic in the Scrum training sessions that I’ve held, and I can’t say that it has magically solved anything just like that, but I think the key to any change is for people to understand the need for it – and the first step for that is to talk about the problems surrounding BDUF (Big Design Up-Front, which is sort of the opposite of Continuous Refactoring).

Please leave some comments.

New employer - new Scrum & Agile challenges

2009-03-17T21:38:00.003+01:00

Unfortunately I haven't had much time to write here lately. My excuse has to be that I just started a new job with an employer a little bit closer to home. I'm very excited about this opportunity, as the company I have now joined is more established in terms of its organization, practices and processes. First time really that I work for a company that isn't in the startup phase.

I'm just getting to know the people now, and studying the development and project models, so no new thoughts about Scrum or Agile just yet. Really looking forward to experiencing development practices in a mature organization (that supposedly have already adopted Scrum, in some way).

I'll be sure to post thoughts and experiences about this new Scrum & Agile journey, as I go.

Forget Change Management - it undermines your chances of success. All changes should be free!

2009-02-16T18:40:00.007+01:00

When you commit yourself to delivering a certain set of requirements ("requirement specification", "statement of work" or whatever you call it) by a certain deadline you automatically create a need of ways to manage change – because change brings risk and disorder, and it risks ruining your time-plans that you have spent so much time figuring out. It risks ruining the architecture that your designers & architects spent so much time thinking through, and it risks turning those already completed programming & testing efforts into waste because a feature is made obsolete.

What's even worse, if you're the project manager then you're likely to be the lucky person stuck with the bill and you have to work out who's going to pay: your company or the customer?
It can be a nerve-wrecking job to be the one that have to make that call to the customer and explain that "this recent request will end up costing 5.000 EUR more and it will delay the project with 3 weeks". And if your requirement specification is the least bit vague or ambiguous at this point, you might end up in an irritated argument or negotiation with the customer about him considering the feature part of the “original scope”, whereas you consider it a Change Request. Not really an optimal situation. But under such circumstances of course you will require a clear way of managing these ugly changes.

So what can you do to minimize these risky aspects of software project management? The traditional way to go is to become even more thorough the next time during Requirements Analysis. I.e. you put even more focus on planning and on trying to figure out even more of the details up front. You have longer pre-studies. You do bigger and more detailed design up front. This way you'll be able to give yourself a better picture of the project before you sign off the requirements and set any deadline. In essence, you build your walls higher and thicker around you. You strengthen the foundation for your Change Management Process and thereby make your project less likely to be exposed to Change and the risks they bring.

With the above reasoning change is something ugly that you want to avoid. But wait. Is change really a bad thing?

I argue that change is great. Change is fantastic.

Screw Change Management processes and the administration they bring! Yes, I said it. In fact, I think we need to give the customer all changes for free, always!

Change means the customer has gotten a different or stronger opinion than he had before, and if I don't adapt to it then the customer will be less happy than otherwise. And what is our number 1 objective? …No, silly, not to make money. Our number 1 objective is to make the customer happy - because happy customers are returning customers. And as a consequence we will make money.

The problem here is much larger than the Change Management process alone. The problem is in the entire traditional approach to software project management and software development.

With a traditional methodology you will definately need Change Management - no doubt about it. There are many great project managers and companies out there, I'm sure, who are working with traditional methods and who are excellent at Change Management.

But I think remarkably many of those never thought twice about the software development methodology they adopted. They just chose one that everyone else was using (for that reason: because everyone else was using it) instead of adopting a software development methodology that maximizes their chances of success. At the end of the day, if you're not NASA and you're not working against the FDA with tons of beurocracy and laws to adhere to, then you're much better off using an Agile development methodology.

You have to accept that you can’t foretell the future. It would be very nice if you could, but you can’t. Sorry. And consequently; change is inevitable – you can’t wish it away. You also have to realize that the customer doesn’t know what he wants – he discovers that as he goes along. Consequently; change is inevitable. Furthermore, you have to realize that developers don’t know how to build the software up front – they discover that as they go along Yes, consequently; change is inevitable.

You can sit there and sob and dislike it all you want, but it doesn’t change the facts.

This is important, and a huge difference to traditional assumptions, so I’ll recap:

Unforeseeable sh*t happens that will affect your plans.
The customer doesn’t know what he wants – he discovers that as he goes along.
The developers don’t know how to build it – they discover that as they go along.

Most people will agree that the above is true. So, why do we rely on dusty old project management methodologies that assume the exact opposite??

The first thing you need to do is to stop committing to both scope and deadline up front. Commit to date rather than scope. This is sometimes called “date-driven development”. You achieve it by putting the requested features in one single list that is prioritized by the customer. Allow changes to priorities, as much and as often as the customer likes. But make sure the priorities are always kept unique for each item in the list and that the priorities are strictly followed, i.e. work is carried out in strict order of priority. As a consequence of adopting this approach there will be no such thing as an “original scope” – because that would imply that there was a “before” and an “after” some event (e.g. a “change”). “Feature creeping” is no longer a bad thing. As long as the customer sticks to the one single prioritized list of features he can have as many features “creep” in as he wants; he can replace all the items with others from one day to another. As long as the customer is happier in the end. With this approach there is no such thing as a “scope change” anymore: the scope instead is something that evolves over time by continuous readjustment .

Secondly, you need to stop making detailed plans. Plan for God’s sake, don’t misunderstand me, but just don’t overdo it. Forget about figuring out what exactly will be done, by who, when, before or after what, two-three months or more away from now. Try to ask yourself; are you creating the time-plan the way you WISH things will happen or as you KNOW they will happen? I’m guessing most of the time the answer will be that it’s the former. The effort spent on creating silly wishful-thinking time-plans is much better spent on something more constructive in terms of bringing the team closer to the end goal (a happy customer) – like getting coffee for the developers. Note that I am not telling you NOT to plan. I am just telling you that all plans are wrong and rarely survive more than one minute in contact with reality. So just minimize the amount of time you spend making them.

Thirdly, start involving the customer. Do it early on. And make sure you have as short feedback loops as you can. With a traditional approach, presenting the product to the customer is often scary because you fear there will be changes, and those changes can be ugly. But you have to remember that those changes are a necessity for the customer to become maximally happy. The problem is not the feedback, the problem is when you have a long lead-time between points of customer involvement (and consequently risk having put in a lot of effort into something that is wrong or will change). So, remember to keep feedback loops (very) short, from the very start of the project to the very end. I suggest 1-3 week loops. Notice that this puts a different demand on the amount of customer involvement than with a traditional approach: the customer needs to be involved (much) more, and earlier. If your customer is not comfortable with that then you’re better off with a traditional approach – but you will less likely to succeed, and the project will be much more expensive because you will have to put risk margins in your pricing.

Fourthly, derive a release plan from reality rather than dream one up ahead of time. Don’t lie to yourself or (worse) the customer by claiming you know exactly what you will be able to deliver by when. You don’t know this! And anyone who claims to is a liar. Instead, commit only to time. Then continuously observe your actual progress rate and use those observations to calculate what parts of the scope it is likely that you’ll be able to finish before the deadline date. You work in priority order and have short feedback loops involving the customer that spark “change” often and early. When you reach the deadline date you have the most important things done.

Fifthly, test early and continuously, and always finish things (completely). Don’t leave some parts of testing for later. Don’t postpone code reviewing. Don’t wait with code merges. The sooner you do things the easier and cheaper they are. Started, half-finished work has no value. For every new feature you start without finishing completely, you add to a pile of more or less known remaining tasks. Over time this pile grows to a mountain of unknown stuff that will get you in the end. Avoid this by really completing things early and do all types of testing early. Continuous integration is key here, and automated testing is a helpful tool.

Since change and unpredictability is inevitable, we have to find a way of working where change is not made ugly by processes. Where there is as little waste of effort as possible when change happens. This is what Agile Development is about – a software development philosophy which covers process frameworks such as Scrum, XP, FDD, DSDM and others. But remember: tThis isn’t the silver bullet that automatically solves all your problems, but it brings maximally good odds of succeeding. Software development and project management is still hard, but at least you’re not making it harder than necessary.

If you’re interested in reading more about Agile and Scrum here are some links:
My blog: http://scrumftw.blogspot.com/
Agile Manifesto: http://www.agilemanifesto.org/
Wikipedia about Agile: http://en.wikipedia.org/wiki/Agile_development

Working smarter is better than working harder - How to increase velocity

2009-02-05T15:47:00.012+01:00

"Why is progress slow? Why does everything have to take so freakin' long? You have to push harder! Work more."

This is something that I've heard a lot, especially from certain types of managers. Let’s look at this more closely, and what it means, and what it is an expression of.

I think it is reasonable to assume that it’s not a matter of the manager actually wanting people to work harder, faster or more just for the sake of doing so. No, instead I think (I might be naive ;-)) that it indicates the manager’s wish to have stuff done sooner. "Stuff done sooner", that is key here. Can this really be done without having people work harder/more? Yes! If you work smarter!

The time that something takes to complete (from its very start, to the time that it is ready) is called “Cycle Time”. You can learn a lot more about this if you study Lean Software Development, I won’t go much further into details here.

Let's investigate what factors affect Cycle Time. How “hard” people work is one factor, yes absolutely. At least up to the point where negative stress kicks in and its consequences appear; defects and demoralization, lacks in communication, loss of cooperation and team synergy effects, and many other ugly things.

At this point I could dive into Value Stream analysis, talk about Value Stream Maps and discuss how to apply the Lean Software Development tools supplied by Mary and Tom Poppendieck et al. But I won’t do that now. In this blog post I will limit myself to only the “implementation part” of the development process. You should know that there is a lot more room for shortening “Cycle Time” than what I will talk about below. With Lean and its tools you should go far beyond only the sprints and observe the steps before and after implementation and focus on the whole chain – the entire “value stream” that flows through your business.

So, back to the subject. In Scrum we continuously measure at least a part of the Cycle Time: We measure velocity. The velocity is the speed with which the team finish things, according to your Definition of Done. Velocity is measured in “Story Points”. You can observe it over a sprint (SP/sprint) or over man-days (SP/manday), or variations of this sort. The point is that you measure the velocity in a consistent way, and you record your findings (in something that I call “the Velocity Log”).

If you say you want to shorten your Cycle Time, i.e. shorten the time it takes to finish something, then what you really want to do is increase your velocity. The bitch here is that the velocity is not something that you can tune by itself. It’s not a knob that you crank up and voila stuff goes faster. The velocity is a result of the environment in which you work. By changing the work environment you will change the velocity. It doesn’t work the other way around. Sorry.

I think we generally need to be much more focused on “helping” the velocity up. It is easy to stare oneself blind on a deadline and all the stuff that has to be done by then, dig in and spend all energy on completing those stories. during such times I think we often forget, or at least seriously underestimate, the power of improving the work environment and the effect it would have on the velocity. This is what "Continuous Improvement" is all about.

Now, what does “work environment” mean?

Unplanned items. They happen all the time, during your sprint and are typically part of your work environment. These are the things which you for some reason just have to do, but that you really shouldn’t (because the team’s job is really to work on the sprint backlog). These are those “negative additions/changes” to your sprint. They affect your team’s ability to focus on the sprint commitment. What you want to do is record your unplanned stuff by putting a note in your “Unplanned”-section on the Scrum Board, for everyone to see. Then you need to work on removing the source of those unplanned items. In my opinion, unplanned items is the number 1 opportunity to increase velocity. Find the root-cause of every unplanned item, and consider if you can’t insert a task into your Impediment Backlog about removing that root-cause. Either way, it is the Scrum Master’s and the Scrum Product Owner’s number one priority to shield the team from these things so that the team is allowed to focus 100% on the sprint backlog.

Concentration breakers: Email. What can I say. As a communication tool it is great, but the way email has been integrated into our daily life is not. If you get a “You’ve got mail!” dialog every time you receive one then your concentration will be broken as many times a day as the number of emails you get. Some days I get hundreds of emails. Jeez! If I was a developer today I would consider not checking my email more than once or twice during the day, and have the client shut down in between. In a Scrum team we rely on face-to-face communication more than written stuff anyway, so believe me, the really important stuff will reach you anyway without you automatically checking your email every minute. And get used to the fact that if you want someone something, then you need to walk over to them and speak to them face to face instead of writing an email.

Concentration breakers: Skype. It is indeed a great tool. In our company we rely a lot on short-lived Skype group chats that we create for various purposes, and we use it a lot for internal communication. It’s a great way of communicating and cooperating. But it can also be a huge concentration breaker. Skype announces new messages by popping up a new window (in the background), with a system tray popup and/or by coloring the chat window taskbar button orange. So every time you receive a message (personal or in a group chat) your concentration will be broken. I guess you learn how to ignore this after a while, but I’m pretty sure that ability is individual. My advice: be careful if you’re using Skype. I suggest killing it altogether and rely on face-to-face communication instead. If you decide to go that route though, make sure it’s a team decision and not enforced as a policy or company rule.

Concentration breakers: Background noise. We work in an open office landscape with very few closed offices. Most of the people sit next to each other in that landscape in order to promote cooperation and communication. The clear drawback with an open office landscape is the noise level from people talking, writing, from computers buzzing and so on. If you’re a manager reading this: don’t be a cheap moron. Invest in sound absorbing cubicle dividers, even if they are more expensive than regular ones (or compared to not buying any dividers at all). If there’s one thing that directly affects the concentration of every single person in the open office landscape – all day, every day! – it is the background noise level. Yes, we can always ask everyone to talk quietly and step into a meeting room for those longer discussions. But also consider: how much are you crippling freedom of communication (and cooperation) by not buying those sound absorbing dividers? It is a low cost for ensuring optimal work environment! While on the subject: seriously consider installing roof- and wall-mounted sound absorbents too.

Unnecessary manual work. Stop and reflect: are there any tasks/actions that you perform often which could be partially or completely automated? Even if the effort of automating it is somewhat large (maybe several days or even weeks of work to build automation or refine tools), it might be worth it in the end. Is it only you that have to do this manual task every time? Maybe it is every developer. 5 minutes a day, for every developer in a 25 person organization, equals 45 man-hours every month. Consider automating many of those manual tasks, and optimize the efficiency of those that you can’t completely automate. If you think about it I’m sure you can come up with a bunch of things. Have a workshop. List stuff and prioritize. I’m sure your Scrum Product Owner will listen if you run the numbers. If you have tips on tasks that can be automated, please feel free and add them as comments to this blog post!

Technical debt. The higher technical debt you have the longer things will take. The system is more complex, harder to test, things are harder to implement, and so on. Addressing technical debt is a subject on its own, which I will come back to in later blog posts. For now I'll just advise you to keep in mind that it directly affects your velocity. You should do what you can to (at the very least) immediately stop increasing your technical debt even further, by letting it take longer to do things "right". It is a short term investment for a long term gain. The bad approach is to do it the quick and dirty way. Remember: the dirty remains long after the quick has been forgotten.

So, if you think progress is too slow, consider working smarter instead of harder. It has a much bigger impact on your cycle time than crunch and overtime has. I really hate crunch. And in so does everyone. Trying to continuously improve the work environment and thereby increasing velocity should be way up high on your agenda! That is how you get things finished sooner, without killing yourself and without sacrificing quality.

I would appriciate comments on your tips and tricks and experiences regarding how to increase velocity!

Updated backlog manager tool in Excel

2009-01-23T16:25:00.005+01:00

I have updated the Backlog Manager tool now, it's a bit simplified compared to last time. Changes since previous version:

- It works in Microsoft Excel 2003 now (previously only worked in 2007).
- You may now freely delete and insert rows in the backlog without risking that the list gets screwed up (due to missing formulas and such).
- There are no hidden columns or weird formulas in the Excel.
- The Card Generator no longer uses the ID-column, but instead the Autofilter result, so you can now filter with full flexibility and the Card Generator will generate cards for the visible rows only.
- The New Story dialog is simplied and looks better :-)
- The index card template looks better and is somewhat simplified.
- Some redundant columns were removed (simplify ftw! :-))

I have updated the old post from September last year so it links to the new version of the Backlog Manager tool / Excel instead.

Anyway, you can find it here too:
BacklogManager-v8.xls

Good luck!

User Stories from the trenches

2009-01-22T15:36:00.012+01:00

I couldn’t resist the topic. But for the record, credits to Henrik Kniberg for the “…from the trenches”-bit :-).

The past few weeks have been hectic, and I haven’t really had the time to post anything new in a while. The reason is that I’ve been in a phase here where we’re ramping up a project and I have been heavily involved in trying to form an initial backlog with an initial set of reasonably meaningful User Stories for that project. I.e.: we’ve been involved in getting a (very large) project “vision” down into a set of stories.

Our traditional method for this was to have the Product Manager (Customer) produce a written description of each feature that he wanted, and then he would hand that over to the Development Department where we (“we” being the team and the Scrum Product Owner: as you may know we use the somewhat controversial “Product Owner Proxy” construction in our organization) would take ownership of that written “spec” and break it down into User Stories that were prioritized and later estimated and delivered in priority order.

My problem with that approach was (and is) that the Product Manager usually (for whatever reason) spent alot of time writing these “functionality specs”.
I have felt that this goes against my core attitude/rule-of-thumb “Never document more than that you have to ask to understand” and that it is pretty "un-scrum". I’ve been wanting to find a better and more Agile way of moving from “The vision inside the head of the Product Manager” to “Something that the team understands, which can be prioritized and brought into the Sprint Planning meetings”.

In this case, with this development project being such a large one (a new product, with so many functions), it would simply have been an overwhelming task for the Product Manager to rely on our traditioinal approach and write all those "function specs" up front – at least considering the usual level of detail in those spec documents.

My idea was to skip the “detail specification” step alltogether and jump straight from the vision to User Stories, and thereby save time by having the Product Manager produce User Stories directly which later could be prioritized and further broken down. This would somehow force the Product Manager to skip details and thus not try to think of everything up front. Oh, and of course, we’d be forced to rely much more on face-to-face communication between the team and the Product Manager, which would be great: “two people by a whiteboard” is always a many times more efficient method of transferring information than by document, right?

So that was the plan, and this is what I have been struggling with for the past weeks (plus celebrating Christmas and New Year :-)).

I can’t say that it went great maybe, but I'm still pretty satisfied with the result so far. Anyway, I wanted to take this opportunity to really reflect over how things went and try to gather my own thoughts about it and nail down some lessons.

Here are some of the challenges with the past few weeks;

I needed to find a way for the Product Manager to “empty his head” into something that could be used as basis for formulating some fundamental stories.
I needed the Product Manager to understand what User Stories are and why they should be formulated in a certain way, what level of detail they should have, and so on.
I needed to find a way – a method and tool – to document the User Stories which would fit nicely with the tools we use further down the road: we use the Backlog Manager tool that I have made available for download on this blog in a separate post.

I approached this by introducing MindJet MindManager to the Product Manager. I know there are open source and other free alternatives out there too, but I haven’t tried any of them myself.

I use MindManager for many different things, and I find it great for emptying my head down into symbols, boxes, notes and relations. So the idea was that it could be used by the Product Manager too, to get the rough set of functions and other thoughts and notes down into a visible map. That map could hopefully then be used to look at to formulate User Stories; e.g. by studying a subtree and all the nodes there to remember what needs to be covered by the backlog.

The map ended up looking something like this, but much bigger (the map below is just a very small dummy example, obviously):

In retrospect, we experienced a few major drawbacks with this method of using a mindmap to describe the vision of the project and have the Product Manager use it as basis for formulating stories;

Coping with keeping the map up-to-date with the stories: as stories are formulated, iterated, thought about, discussed and reformulated, after a while there are changes in the perception of how things are supposed to work. We failed at keeping the map up-to-date throughout this process. After a while the list of user stories conveyed a different picture than the map. However, I think it might be ok that the map is a “consumable” that can’t be trusted after a while; it’s just one tool used along the way and is not really an artifact by itself.
Ones stories were formulated, they ended up being too small. The initial backlog ended up containing much too many stories and each story gave just a too fragmented view of what should be delivered. The symptom of this was that it was very hard for the team to grasp the project as a whole, even with a lot of communication with the PM. The breakdown done was simply much too detailed and too early.

I think my main lesson here is to try to get the Product Manager to just summarize each desired feature as a Theme or Epic (see separate blog post), rather than in a whole bunch of small stories. If there are important details about the function that the Product Manager wants to mention he should write keywords as “notes“ to the story, and not try to write each as a separate story.

Also if there are things in the Epic/Theme which can be given different priorities, obviously those can be broken out as separate Themes/Epics. The ambition should however be to not break everything down in detail.
The finer breakdown, into “ready-ready” (see separate blog post) User Stories should be done by the team in cooperation with the Product Manager – not by the Product Manager himself.

So to summarize on the lessons learned so far:

Get the Product Manager to use a mindmap to empty his head into, with regard to a certain vision of function(s).
Get the Product Manager to look at the map and its subtrees and use that to write Themes (or Epics), formulated as User Stories (“As a I want to so that .”.
Get the Product Manager to write those Themes/Epics in the same or similar tool that you will be working with in Development (e.g. if you can, directly into an empty Backlog Manager excel, if that’s the format you’re using).
When the Product Manager feels he’s covered the most obvious and/or most important Themes/Epics, bring in the team. Let the team and the Product Manager together work on the finer breakdown – of the most important Themes/Epics.
Talk alot about the Themes/Epics: between Product Manager and team. Let the Product Manager describe each Theme, if you have an updated mindmap you can use that as basis for a presentation about each, as part of the breakdown process.
Break down the Themes/Epics one by one, and work face-to-face. Even if it takes the PM and the team a long time, I have a feeling that it is time saved in the end: because the team will get an excellent view of the backlog. A challenge here is to heavily moderate discussions about how to build the stuff: solutions shouldn’t be discussed at this point. That’s done when the stories enter the sprints.
Don’t attempt to break every Theme/Epic down into smaller stories from day one. Make sure they’re explained and moved into the product backlog, but break them down in priority order: start with the most important ones. Leave the lower priority Themes/Epics for later sessions. Just because they’re big it doesn’t mean that you can’t estimate them. Remember: rather roughly right than precisely wrong.
Come back to the bachy hklog after every sprint and review the Themes/Epics and let the team and the PM break them down once they surface in the backlog.

Here is an example of a subtree which could be used as basis for writing a Theme and later for discussing around as part of breaking it down into smaller stories;

Example of a Theme for the above subtree: "As a Administrator I want to be able to manage users so that I can control who has access to what."

Once the team and the Product Manager break it down together, they'd probably end up with something like the following user stories:

As an Administrator I want to be able to add new users to the system so that each user can have his own personal account to login with.
As an Administrator I want to be able to search for users and and be able to sort the resulting list in various ways so that it maximally easy to find what I'm looking for.
As an Administrator I want to be able to Clone an existing user so that I create a new one with the identical data pre-filled so that I don't have to retype everything.
...and so on.

We’re still not done here and have some distance to go before we can say that the backlog is in an “OK” shape. I’ll revise this post and/or make additional posts with more thoughts and experiences as we go along.

Oh, by he way. I always appreciate feedback about your experiences. If you have any clever ideas or thoughts about this or how you work with it in your organization, please leave me some comments.

User Stories, Themes and Epics

2009-01-22T15:31:00.002+01:00

In a coming (much longer) post about story formulation, I refer to the terms “Epic” and “Theme” and I just wanted to clarify what that means to me.
I figured I might as well make it a separate post since it’s somewhat a topic of its own.

A “User Story” is a backlog item. nothing strange there, its everything you think it is. Its format is “As a I want to so that .”.
A Theme (to me at least) is a large User Story.
An Epic (to me at least) is a VERY large User Story.

For example, if you’re building “user administration features” your typical User Story, once it enters a sprint, might be something like “As an Administrator I want to be able to add users so that I can give a person an account to login with.”. That could be the “normal” size of a story that is far up in the product backlog, i.e. close to development. It’s pretty small.

Earlier in the process (further down in the backlog) it could be represented as “As an Administrator I want to be able to administer users of the system so that I can determine who has what access to the system.”. Notice the difference. This is a Theme. It’s much bigger and much less detailed than the story above. But it’s still not massive, and with some discussion it’s possible to figure out roughly what reasonably is hiding behind that formulation.

Earlier still (even further down in the backlog), the formulation could have been something like “As a System Owner I want various administration features so that I can control the behavior of the system.”. This story could be considered an Epic because it is so large and so generic and loosely formulated that it can cover a huge amount of different things.

Kings and Servants - collaborate on the highest priority item

2008-12-05T16:09:00.004+01:00

Some time ago I heard a great suggestion on how to view the idea of a team “swarming” on the top-priority item of the sprint backlog – i.e. how the team should see themselves with regard to who works on what story. As you know, following the priority order is a key mechanism in Scrum affecting your ability for timeboxing (having a flexible scope).

The tip was to view the people who work on the topmost priority item in the sprint backlog as “Kings”. Everyone else (consequently working on lower priority items) are Servants. All the Servants help the King(s) immediately as soon as the Kings need anything. Also, everyone wants to be King.

I’m working on spreading this idea in our organization. I had barely finished speaking to one of our teams before I found a printed picture of a crown put up on the Scrum Board alongside the top priority story :-).

Here's an image that I found with Google that you can print and put magnetic tape on the back of and use on your own Scrumboard(s).

Done or Complete? How to develop in iterations.

2008-12-02T09:19:00.002+01:00

This topic is closely related to that of “Continuous Refactoring” which I am constantly nagging about. It is also related to “Definition of Done”, which is another favorite topic of mine :-).

When are you “Done”? The right answer is: When the Story meets the definition of done. Let’s say your Definition of Done says “Releasable”. So, the Story is done when it is Releasable. Great. But, wait… What does that mean, “Releasable”?

Coders are creative people. I touched this subject in my previous post “1-day remaining forever”. We want to do an excellent job, not just an average one. We want everything we ship to be just right, because bad solutions or sloppy code reflects directly on us and our abilities and creativity. So it’s easy to get stuck in the “almost done” mode, coz we want to polish that extra method or just fix that last design flaw or just rewrite that last class which was poorly written two years ago... So when faced with the goal line (i.e. the point when we’re “Done”) we really want to go to lengths to make sure everything we’ve done is really “ready” and fully thought through, and future-proof, and so on.

But the point I am trying to get to here is that there is a difference between “Done” and “Complete”. Every story needs to be Done in the sense that it fulfills our Definition of Done. But that does not mean “Complete”: it does not mean e.g. “future-proof” or “with architectural perfection”. Those states are achieved over several stories – over several iterations. This is why we call Agile development Iterative. So, think of it like this: “Done” is something we do with each story and each sprint, and “Complete” is something we reach over several stories and several sprints.

I’ve seen it several times; developers dig down into the system and try to solve problems that are of much less priority than the story at hand, e.g. by designing or preparing for some story that is much further down in the product backlog. This behavior builds dependency between stories, it breaks prioritization and it adds a lot of insecurity to the estimates. The proper approach is to focus on the immediate story only, and effectively staying within its boundaries. Trying to stay within the boundaries of a story requires discipline and (not least) a good understanding of Scrum and continuous refactoring.

To sum things up: Aim for “Done”, not “Complete”.

Until next time..!

Flexible scope - A magic bullet?

2008-11-25T18:22:00.008+01:00

Everyone agrees that you can’t both have a rigid deadline and a rigid scope. We’ve been there, Done that, Got the g***amn t-shirt. Software Development is not mass production and last time I checked I wasn’t able to foretell the future. We want to avoid projects where both the scope and the date are fixed. That’s one of the first lessons every developer learns… And that is one of the key things that Scrum brings: you develop in iterations, you timebox and you ensure you have something to deliver all the time, especially release day. Yay!

Timeboxing means the date should be regarded as fixed. Wow, there’s an easy sell to upper management…! But as a consequence we have to allow the scope to flex… Simple enough? Well, allowing the scope to flex might not be as easy as it seems. “Flexible scope” as a concept is easy enough to understand, but it’s not something that happens by itself just because you say “we do Scrum”, develop in iterations with beautiful sprint planning sessions and sprint reviews.

I’ll illustrate with a scenario that I’ve seen in reality several times:

Let’s say you’ve started using Scrum, and one of the key selling points to upper management was: “In Scrum we never move the deadline, we promise to try to always have something releasable at the deadline. I can’t promise exactly what, but we won’t ever need to move the date”. Good start!

So, now you start working. The team is working on a release of (let’s say) two major features (each consisting of several smaller “parts”). The Scrum Product Owner and others work on breaking down the scope(s) into User Stories. But they’re not really familiar with Scrum and the stories they produce are poorly formulated, and are done without them really understanding what a User Story is and what to think of.

Still, eventually a product backlog is ready. But, again since the Scrum Product Owner might not be that familiar with Scrum there is not much effort or thought put into the prioritizations of the stories in the backlog. Anyway, the team starts their sprint with a sprint planning meeting and grabs whatever is on top of the backlog.

Once rolling, the team execute the sprint with little or no regard to the priorities. Everyone grabs a story each and start working.

Work continues along those same lines sprint after sprint. And stuff actually do get done. Progress is OK, people feel OK. So far so good?

Now, when you approach the intended release date, you look closer into what’s been done so far and you conclude that there are still several critical things that aren’t ready yet, or that maybe haven’t even been started yet. In addition, you conclude that there are critical tasks remaining for each of those stories which you previously thought were “Done” – tasks such as integration testing, string translation, code reviewing or similar. Ouch.

At this point, what are your options?

The "easy" way out is to postpone the release date, i.e. effectively breaking the timebox rule. …Scrum breaking down. Oh misery…! And you had such an easy sell to upper management with the fixed-date thing... And all you do is move the date and then have the same problems once more, but that time you've sortof used up your goodwill and no longer have the luxury (?) of being able to postpone the date...
Crunch! There's that word again... But you think "Just this time… This release was special...". You make up excuses about why you’re where you are, and call for overtime. Have people come in a little earlier and stay a little later. Have them come in those last few weekends. Not to mention those last days before deadline. So, the team works overtime, they put in extra hours, and not surprisingly quality suffers, people suffer, shortcuts are taken everywhere… Anyone recognize this?
Flex the scope by lifting out the entire feature from the release. Since there are still critical stuff remaining you don’t have the option to just abandon parts. All you can do is lift out the whole feature. That doesn’t give such a good impression of the whole “flexible scope” mechanism, now does it? “Yeah well, in Scrum we don’t move the date, so we have to let the scope change, and that means removing a feature from the release. We have to allow that. That’s Scrum.”. A job well done. No, this is not what “flexible scope” means.

Those are really the only three options. And what’s even worse is that this is probably not discovered until very late. With only days or a few weeks (best case?) remaining until the scheduled release date… This is Scrum failing big-time!

So, the point I want to make here is that we need to be very careful so that we really maximize scope flexibility. The main mechanisms in Scrum which are affecting this are:

Strict prioritization
Ability of adhering to Definition of Done == Releasable.
Skillfulness of breaking down a scope into user stories

To me “strict” means that we both prioritize AND stick to it, by actually working on things in priority order. If we don’t have a prioritization we can’t really say that we have done all in our power to have everything “critical” or “most important” (or whatever value you want to put in the prio) ready by the time the timebox ends. If you’re not careful, the “completed” part of the scope may be any combination of important and less important things, and (what’s worse) the same is true for the stuff that isn’t done yet.

Prioritizing the backlog needs to be intelligently done, and only 1 person should be allowed to have the final decision about a story’s priority: the Scrum Product Owner. I recommend thinking of the importance in terms of “Business Value”. But be careful; it doesn’t only mean value as in dollars and cents (or Swedish Krona and Öre ;-)). The keyword here is Business. Think of it in terms as Value to your Business, including your own internal business of developing stuff. So things that for example are “architecturally significant” (I’ll post an article separately about this topic, some day) might get a high Business Value because it is important for the architecture and the consecutive development of other stories - even though the story itself may have less of an obvious value to the end user, the customer, your accounting department or your shareholders (but remember: if it is architecturally significant it is most likely valueable to the customer and end user too, it's probably just a matter of thinking about it hard enough and being able of formulating it in words).

The Definition of Done applies to how you “complete” stories. I’ve posted entries here in the past about this and I won’t go into details here. I just want to point out that adhering to the Definition of Done affects your ability of flexing the scope, because you ensure that you stay in control over what’s remaining; you stay on top. The things that have been done are really “done” and are really releasable, there are no hidden piles of work under the rug. I hate surprises and Definition of Done is my best friend :-).

A largely contributing factor both to your ability of adhering to the Definition of Done, and to your ability of following a prioritization, is how good you are at breaking the scope down into User Stories. Don’t regard the scope as a big monolithic lump of functions (e.g. the waterfallish term “Requirements”). Instead, you need the stories to be independent, finishable, valuable and estimatable. A good guide on the way of thinking you want to have is the I.N.V.E.S.T. acronym (stories should be Independent, Negotiable, Valuable, Estimatable, Small and Testable). Writing stories is a skill that you have to work actively with and continuously try and improve.
Don’t be alarmed though; it’s not rocket science. Some breakdown is better than none, and you shouldn’t make such a big deal out of it. Learn by doing. Try different approaches to formulating stories, try the INVEST acronym, try other ways. As long as you understand that the “quality of the breakdown” will affect a lot.

So, to wrap things up: with intelligent scope breakdowns, by following the prioritization and the Definition of Done, we can be more than reasonably sure that the e.g. 85% of the scope which is “ready” when the timebox ends, is really ready, and the remaining 15% is not half-started and we don’t have dependencies between the ready 85% and the remaining 15%. We know we can release something – and that something is actually something valuable (hopefully even the maximum value).

There. Long post this time. But I like this subject. And it's an important one. I’d be glad to learn about other peoples’ experiences. Cheers! My food is getting cold.... ;-)

Have you ever looked at a burndown, really?

2008-11-19T11:02:00.008+01:00

The topic today is about burndown charts. I’d like to make some remarks about what it is used for and why we have it.

First of all, the Burndown Chart is a pretty obvious artifact of Scrum. If you use a wall or whiteboard as your “Scrum Board”, you probably have the burndown close-by and you probably (you should) update it daily. As with most artifacts and mechanisms of Scrum, the basics of the burndown chart is pretty easy to understand. So, at a glance, the burndown chart is self-explanatory.

But is it?

Introduction to Burndown Charts
Let’s start out with discussing what the burndown chart is, mathematically/practically; The burndown is the sum of remaining time for all those items in the sprint backlog, spread out over time along the sprint. So along the X-axis you have your timeline, typically measured in calendar days (at least work-days). Along the Y-axis you have remaining time. You can measure that in Days, in Hours, in Gunships or whatever you want. I recommend that you measure it in (people-)days, though. So, on the daily scrum (the standup meeting) the team goes through all the items in the sprint backlog and they figure out the current remaining time (e.g. by summing up all tasks, or whatever level of detail you have on your scrum board). The sum is plotted in the burndown chart. The result is a plotted line that (hopefully) moves from the top left corner, to the lower right corner. By observing this line the team and everyone else can clearly see how well “on track” the team is: will we reach 0 by the time the sprint is ended?

Can the burndown be "ugly"?
Another important remark (that maybe and unfortunately isn't so obvious) is that the burndown chart is intended as a tool for spotting problems early (by displaying progress daily), which means that it has to be reliable: it has to be the real progress, the hard cold truth - not some made-up half-truthful data that makes the chart "look good". Ever thought of a burndown as "ugly"? I know I have. Sorry, but it is not the burnchart that is ugly (don't kill the messenger): it's the reality. It's the context in which the sprint is executed that is "ugly". As soon as you start "dressing up" the burndown you start hiding the truth. And who will end up biting the dirt for it in the end? Yes, the team...

So, a precondition for having any use what so ever of the burndown chart is that it is as truthful as it is current/up-to-date.

A deeper use of Burndown Charts
But why do you do this at all, track progress and plot it in a burndown? Why display it in this way? Or more specifically: what do you (you, the team, the scrum master, everyone) do with this information?

I know for my own part that I’ve observed hundreds of burndowns by now, and so have all of my colleagues. But I’m not sure that everyone has really spent time reflecting over what the burndown charts mean, other than the immediate, most obvious: that we can see if we have more work remaining than what will be reasonable to finish by the end of the sprint. So, that’s why I wanted to bring this topic up: to get a chance to reflect over how to use the burndown in a deeper, more analytical way: more effectively.

Also, I think most people just reflect over the single, immediate burndown infront of them, without specifically observing or thinking about the burndown in the context of other burndowns from past sprints.

The burndown, in combination with the Scrum Board is an excellent and powerful tool that tells a lot about the immediate climate and environment of the team and the ongoing sprint. If you have knowledge about burndowns and velocity of past sprints, then you have even more powerful information. The burndown – and a set of several burndowns from past sprints – is like a flashlight revealing problems. As soon as there is a problem with progress it is shown in the burndown. But equally important: if you have problems with planning, with product backlog maintenance, with outside disturbance or even with your Scrum implementation as a whole, it will show in those burndowns – if you just know what to look for, and if you realize that this is possible.

So, I guess my advice is: take time and reflect not only over the current burndown, but also reflect over the last 2-3-7-whatever sprints and their burndown, at the same time. I suggest you take a photo of the sprint burndown chart at the end of each sprint and archive it together with records of the estimated and the actual velocity. Then bring that out now and then, let the team get their hands on it and spur discussions about how it looks. Discuss patterns and findings.

Scenarios to reflect over
Below are a couple of made-up burndown charts that I made in Visio (yay!) to visualize some key points. I intended them to represent two consecutive sprints each.

Two sprints showing an interesting tendency...

What would it mean if you see a pattern like above? Here, already from the first few days of each sprint, it is possible to see where the burndown is headed. It is however left like that for too long time, but at least the team acts at some points, by (probably) removing stories from the sprint. So that’s good. But the next sprint shows the same tendency. What if this happens a third time, and a forth? It is clear that the team is over-committing every sprint. I.e. they are not following their actual velocity carefully enough. They are ignoring reality (velocity) and convincing themselves that “next sprint it will be different” – but it never is. So, by observing the burndown over a few sprints you can see that (1) the team acts by removing stories (great), but (2) the team keeps over-committing. It is shown in the burndown as well as in the actual compared to estimated velocity.

Two sprints that show flatlines...

What about the pattern above? Well, the team is unable to complete items – for some reason. It is not resolved (or new problems with the same result keeps appearing) over several sprints. So, the obvious issue here is that the impediment perseveres and renders the team unable to finish items. The not-so-obvious problem here is that nobody seems to act to resolve it. Two sprints is a long time. Remember that Scrum says that impediments should (optimally)be removed in a day. So you really have to issues here that you need to resolve: (1) fix the impediment stopping the team from finishing items, and (2) find out if and why nobody is addressing that impediment.

The two examples above are easy ones, but still I’ve seen them both. I think a lot can be saved by observing both the current burndown and really reflecting over it, but also (not least) by now and then observing the burndown of several sprints in a row.

Pair programming according to Crisp

2008-11-10T19:13:00.001+01:00

Read a very interesting article on Pair Programming today, written by Jan Grape of Crisp. Sorry it's only in Swedish, but here you go.

Allow slack, don't allocate 100%

2008-10-30T16:05:00.003+01:00

A classic approach to resource allocation is to aim for 100% allocation. Anything else would be cost-inefficient, right? Having people sitting around doing nothing can’t be a good thing, right? …Well, wrong…!

Aiming at 100% allocation would be desired only if it was possible to know and plan every detail that every person has to do, in advance. And you can’t. Allocating less than 100% doesn’t mean that the unallocated time is expected to be spent on youtube (no offence, youtube is great! ;-)). No, it means that the unallocated time is expected to be available for all unplanned things that has to be done in order for everyone to do their planned work and keep the pace up.

It’s important so let’s repeat it: You should allocate less than 100%. The remainder, let’s call it Slack, is expected to be available to do useful things that can’t be planned for. What are those “useful things”? Well, helping your teammates for example. If you don’t allow for slack the team members won’t have time to help each other. So the team’s productivity will actually decrease because the cooperation effect is lost.

Usually the purpose of trying to allocate to a 100% is to make sure that productivity is maximized. But, in fact increasing allocation towards 100% will risk giving the opposite effect.

Obviously, the trick is to have just enough slack.

So, slack is good. You want slack, and you need to stay weary of any attempts or implied desires to the decrease slack time.

There are several methods you can use to ensure you introduce the slack time as a mechanism in your planning:

Continuously measure and use “Focus Factor”. A typical focus factor might be somewhere between 0.6 and 0.8 I would say, but there are no rules and differs for every team. So don’t focus too much on this number: it’s a crude measurement and it has much more to do with how large/many impediments and how much disturbance the team has, how good your user stories are and how good your backlog is, than e.g. the team’s efficiency.
Expect no more than e.g. 6 hours of "available work-time" every day. We’ve been experimenting with values as low as 5.5 and up to 6.5 or 7. If you use a project tracking tool it should be possible to set “Working hours” somewhere as a global setting for your project. Either as a percentage of a working day, or as a start- and end-time of each day (which was the option in our case).
Measure actual velocity and compare to actual, continuously and let the team help out to give feedback on their sense of “load”

We’ve tried all three, and I think a combination of them is the way to go, depending on situation. Key is to understand why you need slack. If everyone understands that, then you’re able to use either method whenever it feels right.

For another, shorter and more pedagogical way of saying what I say above, I recommend Henrik Kniberg’s slide that you can find here (see page/slide number 26): http://old.crisp.se/henrik.kniberg/presentations/2008-10-13-HiQ/Introduction-To-Lean.pdf

Fixed price contracts: Money for nothing, Change for free

2008-10-28T10:21:00.003+01:00

Just a tip: If you haven't seen or heard of it already, read this recent blog post by Jeff Sutherland about the concept of "Money for nothing, Change for free" which is his name on the approach of setting up development contracts; Agile Contracts.

I haven't applied this approach myself, but it is really a fascinating topic. I come from the consulting business so I've experienced first-hand the horrible reality of Waterfall classic fixed price contracts. I'd be very interested in trying to apply Agile and Scrum to such a situation...

Bug-fixing inside inside sprints or outside?

2008-10-23T14:38:00.004+02:00

Today’s post is about how to manage bugs.

First some background. We’re a product developing company and we maintain our own products as they go live. A lot of our work focus on new feature releases for the existing products. We have continuously updated lists of live bugs that has to be dealt with.
Even if you’re not a product developing company like we are, I’m sure that you can relate to the situation.

Ok. Let’s take a minute and look at what a “bug” is, and its difference to “a story”. A bug is something that is often unestimateable. Trying to estimate it may take longer than actually fixing it. Or estimating it may even mean fixing it. A “bug report” is a description of an unwanted behavior of the system or, not seldom, a question about a possible unwanted behavior. Anyway, hopefully it also has some type of “priority” relative to other bug reports (and hopefully there’s a single person who prioritizes – or you’ll have a mess). Of course, bugs are unwanted and we need to strive towards continuously improving so that we avoid bugs altogether. The day we have zero bugs I’ll erase this blog post ;-). But until then we need a way to deal with them.

Since recently we use Trac to manage our bug lists. We used to use Hansoft. We moved to Trac because we felt it was more suited for our needs, and it has an excellent and highly configurable web interface. It is also possible to integrate to Subversion, which is the version control system we use.

Now to the options: Do you bring those bugs into the sprint backlog during sprint planning? Or do you leave room in the sprint backlog to be able handle bugs during the sprint, and maybe even set aside time and resources for it? Or do you have “Bug-fixing Fridays”? Or do you have dedicated teams for it? Or maybe you have “Bug-fixing Sprints”? The options are many, and we of course haven’t tried all. I’ll try here to summarize some of the most obvious methods that I can think of though;

A team of dedicated resources work in parallel to the development teams
This has been the approach in one of our products, for almost a year now. We have two dedicated full-time developers who work with bugs. It works great but it requires these resources to be highly multi-talented. These guys don’t work in sprints (at least not with the bug fixing) and they set aside time for the bug fixing activities; on a few specific days of the week they work with bug fixing and release patches to the live product. The other days these guys work with other “urgent” things which we want to shield from the development teams – much of these things have to do with supporting other internal functions such as sales, advertising, marketing, customer support, etc. We call this team “Client Services Team”.

You’ll need to be careful so that these guys are kept happy. If they want to circulate into development you should try to accommodate for it. Working with bugs can be tiring and stressful. Another downside to this is that the developers who originally cause the bugs seldom get to fix or even see the problems so the bugs doesn’t really serve as an incentive for them to continuously improve. On the upside though the release cycles are short and the bug fixing guys often see immediate result of their work. Also, they get to see and learn to work with a huge variety of technologies and parts of the system.

Dedicate an amount of time during each sprint for bug-fixing
This is our current approach in another one of our products, for a couple of months now. We (most often) do 1-week sprints, so the cycles are short anyway. Not sure this approach would work if we had longer sprints. But we let the development team assume the responsibility for dealing with bugs. They do this by means of two mechanisms;

Bugs as stories: every week the Scrum Product Owner and the Scrum Masters meet to discuss what’s currently critical or high urgency in Trac, and appropriate issues are taken out and put into the product backlog as stories. They attempt to (guess)estimate them using the StoryPoint value (Gnome Days, see other blog posts below). These stories are merged into the backlog just like any other story, and the prioritization is made relative to any other stories in there.
Dedicated time every week for “other” bugs: every week (decided on the sprint planning) the team appoints a person and a specific day of the week which will be used for bug fixing (bugs that are not already in the sprint as stories). On that day, Trac is checked for the current buglist and bugs are fixed from top to bottom. Bugs here are not fixed in order of priority but rather in order of size (smallest first). “Smaller bugs” get done faster, regardless of their priority. The idea is to avoid having bugs in Trac open for ages which are really annoyances but that are not critical. I think maybe this sorting on size instead of priority is a sign of the prioritization mechanism not working properly…

This approach seem to work well too, but it requires that the Scrum Product Owner and Scrum Masters really take time continuously to look at the buglist and update the product backlog properly.

Dedicate a whole sprint every now and then
We haven’t tried this approach. I think there are some problems with it. For one, the team will lose pace and we’ll break the Agile principle of “maintaining a constant pace indefinitely” (see http://www.agilemanifesto.org/principles.html). Secondly, the cycle time from bug report to fix will most likely be longer. This approach doesn’t work well for critical bugs that has to be fixed “right away”, unless the sprints are ultra short. Those critical things would most likely end up leaking in to the development sprints. The upside could be that the developers would get to fix their own bugs. It would also force a wider spread of competence throughout the development team(s) instead of e.g. limiting it to certain individuals.

Bug-Fixing Fridays
We haven’t tried this approach either. It sounds interesting enough though. I'm curious to find out if anyone's tried it. It would mean that you regularly dedicate a certain day(s) for everyone to do bug fixing. Maybe every week, maybe more seldom. The problem with this approach though is that it may be hard to have a feeling of continuity in the bug fixing process. And bugs that take longer than a day to fix will end up open for a long time.

On IDG's article about critisism against Scrum

2008-10-22T17:57:00.007+02:00

Here's a Swedish article from IDG about that "the critisism against Scrum is growing".

It has something of an eye-catching headline, but the article is otherwise thin. The only really valuable thing in there are the statements from Tobias Fors about having to understand Scrum before trying to adjust it. But the topic as such and the angle of the article is just rediculous. Also, I think Tobias was somewhat misquoted or misunderstand with regard to what is written in the article about retrospectives.

The author quotes Ivar Jacobsson about the teamsize and iteration length limits of Scrum being a downside, as opposed to other methods. The problem with that argumentation is that Scrum is then blamed for not allowing "large teams" or "long iterations". You have to stop and ask yourself why. Why do Scrum limit the team size and the iteration length? Obviously because statistics and years of experience in software development projects has shown that those are factors directly affecting the outcome of a project.

That Scrum doesnt suit "large systems that are based on a service oriented architecture or large organizations" as a general statement is gong too far. If Scrum works or not is in my opinion much more a matter of the organization's willingness and capability to try - not a problem with Scrum itself.

Sure, some of the advantages of Scrum are lost if you are strictly controlled by a ruleset that is waterfall-based, such as public fixed price tenders, or if you work against the FDA or similar. In those cases the problem isn't with Scrum. The problem is with the very preconditions for the projects. In that sense this Ivar Jacobsson really do has a point. But for the author to write an article under the topic "critisism against Scrum" on that is to me a sign of news drought.

Traditional roles vs Scrum roles

2008-10-20T15:51:00.015+02:00

Ok, you have an "ordinary" organization with traditional roles, and you want to do Scrum.
You have someone with “Project Manager” or “Producer” written on their business card. Maybe you have someone who’s “Technical Project Manager” too. You have your team, obviously. And last but not least you have some Customer. In our company we do our own products, so we don’t have external customers in that sense. We do have users who are extremely important of course, but I regard them as end-users rather than assigner in the sense of who’s ordering work from us in production. In our case the assigner is the person with "Product Manager" written on the business card – she’s the one making the business calls and the one outmost responsible for the direction and the result of the product. In the world of consulting the customer is obviously the external party who orders and pays for the work done. Anyway, the team does the work. The Project Manager is usually responsible for the high and the low; from high-level planning and risk management down to resource allocation, problem-solving, task assignment & follow-up, etc. The customer makes the order and expects delivery. Below you’ll see a schematic of the traditional roles.

Ok, so we’ve ironed out how things usually look. The above would apply to organizations with external customers (like consulting companies) as well as organizations with internal customers (like us, who own and distribute our own products) – it’s just a question if the Customer is an external or internal role.

Now, one simple approach when introducing Scrum is to put the Scrum Master-hat (SM) on the project manager. He’s the one doing many of those duties anyway, like helping the team move forward, keeping track on who’s doing what, the one controlling the status meetings, etc. Let’s say you do that, so then what roles are left to figure out? Yeah, the Scrum Product Owner. Ok, that sounds simple enough, at least in a company like ours where we already have someone internally who’s “owning the product” and making the business decisions: the Product Manager becomes the “Scrum Product Owner” (SPO hat). Even the name of the Scrum role suggests this construction. Now, in a consulting company, with external customers, I’m not sure it’s as easy as that. I’ll leave that open for now. Anyway, here’s how it would look:

Wow. You now have a team, the person who used to be “project manager” who is now called “Scrum Master” instead, and the person who used to be Product Manager is now called Scrum Product Owner. Sounds alright, right?

Wrong! But what’s wrong with this picture?

A lot of things. One of the most obvious things with this setup is that the Scrum Master isn’t someone within the team. The ever-so-important purpose and work of the Scrum Master is missing or at least the efficiency will be lower because of it. See previous blog post below about why the Scrum Master shouldn’t be a formal leader and needs to be someone within the team. Another factor here, if you have some history in your organization, you’ll need to remember that the project manager has been the one putting the strain on the team, the person who’s been breathing down the team members’ necks. The one with the whip. So it’s a person that the team will most likely never see as “being on their level”; a fact which will make it hard to perform the duties of the Scrum Master to any meaningful extent. So what can you do to change that? Maybe what you want to do is give the Scrum Master hat to someone in the team? Voila, you have your roles. And the project manager is out of a job. Or?

But wait, so far we haven’t discussed the appropriateness of giving the product manager the SPO hat. We just assumed it’s best because the names of the roles are similar :-). I argue that you need to carefully consider whether or not the product manager should really be your PO. In fact, I think that instead the project manager is very suited to take on that role. Because what happens is that the “project manager duties” are smeared out across (A) the team and the Scrum Master, and (B) the SPO. The daily stuff, the things that have to do with who does what, when and why, is handled by the team and the Scrum Master, and the planning, prioritization, risk management and so in is done by the PO. And so, the product manager in this scenario (or in the case of a consulting company – the customer) is perfect for being regarded simply as “Customer”, nothing else.

Now, I know there is some people arguing out there about whether or not “Product Owner proxies” should be used. Here's a link to a Swedish article by Tobias Fors on that subject. As I understand it, the argument from those opposing it is; if the person with the outmost responsibility (e.g. the assigner – the customer) for the product is not deeply involved with the team and instead those duties are taken over by the former project manager – now called the PO – then the PO becomes just a “proxy” for the real product owner. And a key point in Scrum is to involve the stakeholders early, with the team, and get the feedback directly and quickly. Hence, relying on a proxy for it will negatively affect that. So the argument is to forget about proxies and instead let the real product owner be the one with the Scrum Product Owner hat. Well, I have to agree with that. But there’s a reality too, where the product manager is loaded with business decisions, meetings, research, spreadsheets, budgets, administration, etc. In an ideal world I would change all that and remove all those things from the product manager’s back so she could focus on the important parts; what goes in the product – and thereby give the Scrum Product Owner role to her. But we don’t live in an ideal world and I will have to make do with what I can get. And the next best thing is a really engaged project manager (who I call “Scrum Product Owner”), who has the direct link to the product manager (who I call “Customer”).

So, finally, my recommended setup is as follows;

With this setup, some of the advantages are (not in any particular order);

The Customer/ProductManager can focus on other things too, and does not have to spend all his/her time on helping the team get the job done. He/she still has to be around on a regular basis and provide feedback early, at least on or around every sprint review.
The Scrum Product Owner is a full-time role who's job it is to (a) know the customer and (b) know the team. The challenge here lay (among other things ;-)) in how well this person understands what the customer wants, and how well that understanding is reflected in the prioritization of the product backlog and the breakdown into stories.
The Scrum Master is someone within the team, not a formal leader. Also, with this construction the Scrum Master can really focus 100% on being the one sweeping a clean path for the team ("the curling parent"), without any specific further responsibilities outside those of the SM and those of the whole team as a collective.

In an ideal world, if I practically had the choice (e.g. if I were to build up a new organization from scratch and got the power to decide every aspect of it) then I would try and let the "Product Manager/Customer" have the SPO hat. In that sense I do agree with the concerns about "Proxy Product Owners". But if that is not possible - which I don't see that it is in our case, and probably in many other cases - then I chose the pragmatic approach. I think the above is the better than... Well, what's the alternative? To have a Scrum Product Owner or a Scrum Master that for various reasons aren't focusing 100% on their roles.

"1 day remaining" forever

2008-10-13T13:51:00.003+02:00

Let’s say the team is working on some story and need to finish a task which is originally estimated to 4 days. Once the team starts working they continuously decrease the estimate until they reach 1. The task then remains on “1 day remaining” for several days – even though the team actually do work (and hard, too) on finishing the task. You ask the team what’s going on, why the task never reach 0, and the reply you get is “Well, things keep popping up that we didn’t think of which has to be done too in order to finish this task. We can’t finish the task unless we also handle those things.”. Let’s say you don’t only see this occasionally. You see it over and over again, Story after Story and Sprint after Sprint.

What is the problem? (Please leave me some comments below if you’ve seen this problem). I have some likely suspects:

Estimation technique
I.e. your estimation process. If you keep having this problem then you probably need to consider if it is because you keep missing something in the estimation. Maybe it is always the same “thing” that keeps holding the task up from being finished? Could you incorporate that “thing” as an item in a checklist which is always considered during estimation meetings? Do you have the right people attending the estimation meetings? Do you have someone with deep enough, and right, knowledge about the system and technical domain to give his/her input to the estimate?

Technical Debt
As a system ages, the shape of its code, architecture, design etc deteriorates – its natural, and every coder knows it. The problem is that the rate of decay is so much higher if the people working on the system continuously take shortcuts. There’s a saying that I keep coming back to; “The dirty remains long after the quick is forgotten”. Think of that “dirt” as taking a loan. Eventually you will have to pay it back, and in the meantime you will have to pay interest on the remainder. “Interest” in our world is in the form of us having to deal with problems when developing new things; things inevitably take longer because the system is so complex. Paying back the loan will mean having to take time to eliminate those shortcuts (that dirt).

If your team tends to stay on “1 day remaining” for ages then you’ll need to consider if the problems they have (the unknown stuff that appears) are due to technical debt. If it is, you’ll want to prioritize very high some efforts for paying back some loans. There’s a business reality out there so, most often is simply not possible to halt development for six months and have all resources work only on refactoring. But at least you can ask yourself; are we increasing the debt still as we build new stuff? And if so, why? Most likely the developers do not want to take shortcuts, but they are more or less forced to because of pressure (implicit or explicit) from stakeholders. You’ll want to look at your team’s environment and see if you can somehow remove the feeling of them not being allowed to take longer time to do things “right”.

A mechanism to emphasize this is to introduce a bullet to your Definition-of-Done: “Not increased technical debt”. It should mean that it is not OK for a task/story to be finished until the implementation is such that it doesn’t increase the technical debt. I.e. it WILL inevitably take longer to finish, but the shape of the code will be better than otherwise. Next step is to replace that bullet with the following: “Decreased technical debt”. That will mean that with every story that is finished there has been some effort put in to refactor (even if ever so slightly) what’s already there.

Formulation of User Stories
If your stories are formulated in a bad way then the team will definitely have a hard time finishing. Maybe the stories are too big? Maybe there’s too much detail and no room for flexibility? If your stories contain every little detail then you’re really back to waterfall again. You’ll spend enormous amounts of time writing specifications, leaving no room for the developers to think for themselves and no way for them to build it in any other way than what’s specified. And hence, it will be naturally hard to develop in iterations. The problem is that the flexibility of the scope has been lost. You essentially end up with a fixed scope.

“Fixating a scope”, to me, is OK. Wow, that’s a controversial statement. Let me try and explain what I mean. As long as it is done on Theme/Epic level (i.e. something along the lines of what’s traditionally called “Function Specification” level) – and not on Story level (i.e. not like a Requirement Specification), then it is OK for it to be considered fixed. If you have someone (e.g. a customer) ordering something (e.g. a feature or a whole system) then it’s OK for that person to really think through what he/she expects. It’s even OK if that is written down.

But as the feature(s) come closer to the development team then it is important that it (however detailed) is broken down into User Stories which are flexible enough to be implemented in iterations, so that you are able to deliver something working at the end of every sprint.

If you’re the Scrum Product Owner, then spend some time thinking about how the stories are formulated. Is the level of detail “just right”? Is there enough flexibility in there for the team to be able to finish it (i.e. without being extremely restricted to an exact certain scope)?

Badly overlapping User Stories
Obviously there’s a million ways to derive (Mike Cohn calls it “trawl”, by the way) User Stories from a vision of a feature or system. I’ve stated previously that I believe the approach should be to divide the feature(s) into verticals (we call them “slices”). Regardless of if you listen to my advice or not :-), you risk running into the problem with overlapping stories. Note that I do not mean “overlapping” as in overlapping code/subsystem: on the contrary, continuous refactoring is all about iterating and reiterating the same code over and over again. What I mean is overlapping functionality; the problem comes when you can’t finish a Story because it intertwines with other Stories, in a way that you can’t tell which story you’re really working on. That is a badly overlapping User Story.

If the team keeps remaining on 1 day, then you should consider if maybe badly overlapping stories is contributing. Try reformulating stories with this in mind. Can you think of other ways to write the stories? Think of the acronym “INVEST” as a rule of thumb; stories should be

Independent
Negotiable
Valuable
Estimateable
Small
Testable

Finished does not mean Complete
This is, I think, as much a result of all of the above as it is a fundamental attitude in the people working on and around the system. When a Story is “Finished” it doesn’t mean that what has been done is in any way “Complete”. I keep coming back to it: Continuous refactoring, iterative development – this is what it is all about.

When unforeseen problems appear and a team stay on “1 day remaining” for ages, can it be that they put the wrong meaning in the word “Finished”?

“Finished” should mean that the story is implemented in some way. It doesn’t mean that the implementation has to be perfect, and it doesn’t mean it needs to be completed and never again returned to. I’ll post a separate blog some day about whether or not this means increasing technical debt or not (in my opinion it does not).

This, I think, requires a change in attitude among the developers, the product owner and the customer/stakeholders. A sprint demo/review doesn’t mean showing Complete stuff. It means showing Stories finished. “Finished” as in “Done-Done”, yes. But that doesn’t mean that we don’t go back immediately next sprint and add, change or remove stuff in order to finish another Story.

Also, another related cause may be that the sprint is too long. The longer the sprint, the more and the heavier the feedback will be from stakeholders. The more seldom the stakeholders see the result of a sprint the more they will feel that they have to “get all the comments out there” and thereby the bigger deal the sprint review will be to the team. And the bigger deal the more it implies “Complete”. So, consider shortening the sprint length (a lot) and get the customer and other stakeholders involved much more often. It is a matter of building confidence between team and customer – confidence that the result in the end is what was expected (or actually even better).

Beyond the buzzwords

2008-10-09T14:30:00.006+02:00

What is “Agile”? And what is “Scrum”? And “XP”? Oh.. And Lean? What is that? Can you “work with” Agile? Or what exactly is it that you do? Are all just different and c00l buzzwords meaning the same thing?

Oh, the confusion… =)

Well, I think of Agile as a “value package”. The word “Agile” is simply the name of a set of core values and principles about how “development projects” is viewed. So, you can’t really “work with Agile”, but you can have processes and project methodologies that are Agile – they fit the Agile core values. Here’s what’s written on Wikipedia about Agile Software Development: “Agile methodologies generally promote: A project management process that encourages frequent inspection and adaptation; a leadership philosophy that encourages team work, self-organization and accountability; a set of engineering best practices that allow for rapid delivery of high-quality software; and a business approach that aligns development with customer needs and company goals.”. There’s a manifesto that covers the core values, which you can find here: http://agilemanifesto.org/.

So, “Scrum” then, what is that? Is that a process? Or a is it also just a “value package”? No, to both questions. Scrum is an agile "process framework". It is not a process, it is a framework for processes to exist (be formed) within. Scrum is an interpretation of Agile Development. Right or wrong, I view Scrum as an “instance” of an Agile development approach. Scrum sets the boundaries and the general direction for the processes but it doesn’t state exactly what those processes should be or what steps and methods they need to hold. That is why it is said that if you “implement scrum” you have to find your own way: you have to create your own set of processes and practices that fits your organization and the Scrum framework. It is quite correct to say that you “do Scrum”: it means that you have identified and enforced a set of practices that cover all the aspects of Scrum. You can read more about Scrum on (lots of places) for example http://www.scrumalliance.com/ and on Wikipedia.

Ok, and “XP” (eXtreme Programming)? Just like Scrum, XP is an “instance” of an Agile development approach. So XP too needs interpretation and adaptation to be implemented in your organization; it is not an out-of-the-box ready set of processes. The difference between Scrum and XP is that Scrum is more focused on project management and team aspects of software development, while XP focuses almost entirely on the hands-on development practices. Scrum for example talks about how to plan your iterations, using Sprint Planning meetings and Sprint Reviews. It doesn’t say much about how to actually write code. XP on the other hand doesn’t say as much about managing your project (although there is some overlap) but focuses much more on how the developers need to sit together and work in pairs, or that you need to have continuous integration, automatic testing, Test Driven Development (TDD), and so on. Scrum and XP fits really nicely together, and together they form a great and whole-covering framework for a really agile development approach. In our organization we have yet to adopt XP to any extent. But we’re getting there.
You can read more about XP on (lots of places) for example Wikipedia.

And finally, what is “Lean” in all of this? Well, it too is a way of observing the world - the world of manufacturing to be precise. According to Wikipedia, Lean is “the practice of a theory of production that considers the expenditure of resources for any means other than the creation of value for the presumed customer to be wasteful, and thus a target for elimination.”. It originates from the automotive industry (e.g. Toyota in the 1950s) and was later adapted to software development (Lean Software Development) by Mary and Tom Poppendieck. Just like Agile it covers a set of core values, and sets a direction for how to go about “manufacturing”, in our case Software Development. Lean is all about spotting and removing sources of “waste”, decreasing cycle time, continuous improvement, respect for and personal growth of individuals, etc. Lean and Agile fits very nicely together.

Confused? Yeah, me too ;-)

Slices (Verticals), User Stories and Scrum

2008-10-08T10:25:00.004+02:00

We try to divvy our stories in a way that they span from top to bottom; through all layers of the architecture. We call such an isolated part of a function or project, a “Slice”. It is also commonly known as “Vertical”.

But why develop in slices, or verticals, why even care about it, why not just do the User Story regardless and forget about if it is a horizontal or vertical or whatever…? Well, we’ve tried various approaches here. The one thing I am certain of is that this is hard. It requires skill to be able to produce “good” user stories. And not only is it hard, it is also very important: because badly formed User Stories can wreck velocity. Properly shaped and formulated User Stories is one of the basics, a requirement, for a well-functioning sprint execution. So, Hard + Important means that we have to focus on learning it and really spend time thinking about it – and continuously evaluate our approach and view of the matter.

Ok, so then why vertical and not horizontal?
If you do things horizontally, isolated to each architectural layer, then it will be hard for you to actually deliver working, testable code. By doing Slices you are forced to deal with every layer right from day 1. This foundation will then be built further upon, as you iterate the software with every new slice. Also, working vertically – meaning involving every layer in every slice – will often require different competencies (for example UI/web, C++ and SQL/db). Suits nicely with our cross-functional Scrum teams, doesn’t it?

To use Lean terms, it’s about decreasing the “batch size” and ultimately minimizing cycle time; i.e. the time it takes until you deliver something valuable to the customer. If you work horizontally it will take longer until the customer has something that brings value. A complete user interface, but empty of all functionality, is rarely valuable to the customer (and rarely “complete” for that matter, until you’ve built the underlying stuff). And it is definitely not something “releasable”. Remember that you want to strive towards having something releasable every sprint. Sometimes (often?) it doesn’t make sense to release “half a feature” (e.g. “Add User” without “Edit User” functions), but at least you need provide that option and let the customer decide what he wants and doesn’t want: don’t make the decision for him by implementing horizontally.

So what forces are in play here, with regard to stories, architecture and Scrum? Well, let’s recap;

We’ve just concluded that we have a requirement to build vertical increments of the system: Slices.
We also have a requirement to “finish” (Done-Done) things, with each sprint.
We even have a requirement to be able to “Release” at the end of every sprint.
And we have a requirement to think iteratively, i.e. we should come back continuously and refactor our code (continuous refactoring – see blog post below)

Should the Actual Velocity measurement include “Unplanned stuff”?

2008-10-07T12:23:00.002+02:00

A question that keeps popping up here is whether or not the Actual Velocity (i.e. the sum of the Story Point values of all Done-Done stories from a sprint) should include “Unplanned stuff”. I.e. things that were not expected and which are not really part of the Sprint Backlog. The question assumes that the team somehow estimate (in Story Points) that “unplanned stuff” as it appears, and put it into the Sprint Backlog, and consequently track it to Done-Done state and as a consequence include it in the Actual Velocity measurement.

First of all, I think the way to go is to minimize the amount of unplanned stuff. Period. If you have lots of unplanned stuff then you should consider shortening the sprint length, or possibly look at how well the Product Backlog is managed and prioritized.

But there may still be unplanned stuff that appears. There’s no way around it. It’s a reality and we have to be agile enough to be able to handle it.

Let me just emphasize that if you were to estimate the unplanned stuff, you would have to estimate it using the same unit and estimation reference as the regular User Stories; otherwise the Story Point values would not be comparable or possible to sum up. In our case we estimate all our User Stories using “Gnome days” (se blog posts below), so the Story Point value of the unplanned stuff would have to be estimated using that unit too.

So in every sprint we have two alternatives, right: either you do include unplanned stuff (in Actual Velocity) or you don’t. Let’s take an example where we have a team doing a sprint starting out with 30 SP of Estimated Velocity. Let’s also assume that the team so far has had no unplanned stuff appearing in previous sprints and the Estimated Velocity of 30 is pretty accurate and stable.

Case 1: the team decides to include “unplanned stuff” in the Actual Velocity measurement
The team works for a sprint and there’s a bunch of “unplanned stuff” that appears. The team estimates it and includes it in the Actual Velocity measurement at the end of the sprint. The team’s Actual Velocity will probably be close to the Estimated Velocity, even though parts of the work Done-Done is actually “unplanned stuff” that has replaced User Stories that were originally part of the sprint backlog. As a consequence the team will have completed less User Stories than expected, because it had to deal with that unplanned stuff. So the understanding of the team’s velocity at the beginning of the sprint has proven incorrect, in the sense that it incorrectly indicates that the team can perform 30 SP of User Stories from the Product Backlog. That is true only if there’s no new “unplanned stuff”.

Case 2: The team does not include “unplanned stuff”
The team works for a sprint and the unplanned stuff that appears during the sprint is not included in the velocity measurement, so when the sprint ends the team’s Actual Velocity (25 SP for example) will be lower than the Estimated Velocity (30 SP in the example above) – because the team was distracted from the original sprint backlog doing that unplanned stuff. As a consequence, on the next sprint planning the team will observe its past Actual Velocity (25 Story Points in this example) and they will thereby commit to less. This way, they automatically adjust for a “similar amount” of unplanned stuff to appear in future sprints.

Conclusions
Given that you have a relatively steady inflow of unplanned stuff (as opposed to very rare occurrences, or spikes) in your sprints then you do not want to include Unplanned Stuff in your Actual Velocity, because it decreases the quality/reliability of the measurements. As a consequence you risk constantly taking in more User Stories than you’re actually able to complete.

Note that I’m not saying that you shouldn’t estimate or do the “unplanned stuff”. If there’s unplanned stuff that appears that has to be done, then sure you can include it in your scrum board, estimate the remaining time and track it in the burndown. But do not give it a Story Point value and do not include it in the measurement of Actual Velocity.

What metrics should be collected and how to measure and use "Velocity"?

2008-10-06T09:06:00.008+02:00

What should be measured during a sprint? Obviously, you can measure a million different things. My suggestion below is tilted towards the simplistic approach. I like to keep measurements and maths to a minimum. Don’t get me wrong; I love Excel, metrics and complicated statistical formulas... But as far as the team goes, and what they need to care about and focus on, I suggest sparing them the nittybitty details and just collect the bare essentials.

Estimated Velocity: The sum of the story point value of all those User Stories taken into the sprint backlog (at start of sprint). You can read some more about the concept of "Velocity" here, on the Scrum Alliance site.
Actual Velocity: The sum of the story point value of all those User Stories that are determined to be Done-Done at end of sprint.
Average Actual Velocity: This should be the Actual Velocity averaged out over a number of sprints (provided the team i relatively stable – or this will have little or no meaning).
Mandays in the sprint: This is simply the number of persons in the team multiplied by the number of workdays available in the sprint.
Focus Factor: The Actual Velocity divided by the number of Mandays in the sprint.

Example: Let’s say I have a team consisting of 4 full-time individuals and one person working half-time with some other duties. Let’s say they work in 2 week sprints. Let’s further say that the team takes in a total of 35 story points at sprint start. Let’s say they manage to complete (Done-Done) a total of 28 story points. So;
Estimated Velocity = 35
Actual Velocity = 28
Average Actual Velocity = Avg(28) (So far only 1 sprint to average over)
Mandays in sprint = 4.5 persons * 10 days = 45.
Focus Factor = 28 / 45 = ~0.6.

So, what values do you use during Sprint Planning? I suggest the following:

Average Actual Velocity, or at least last sprint’s Actual Velocity, to compare the new sprint’s Estimated Velocity to, as a sanity check.
Probable Velocity = Mandays available in the new sprint X Focus Factor.

Example (continued from the example above): If the team and the sprint lengths remain unchanged then the team should probably not take in more than approximately 28-30 story points into the new sprint. If the new sprint is, for example, 15 days instead, you can use the previously measured Focus Factor to calculate the probable velocity; (4.5 persons * 15 days = 67.5 mandays) * 0.6 FocusFactor = ~41 story points.

The most important thing is that the team feels comfortable with the commitment on the sprint planning. The sum of the stories selected by the team should however be verified against previous velocity to make sure that the team hasn’t substantially under- or over-committed.

A team’s velocity is a measurement of the reality. It can’t be “too low” because the reality is what it is; making up excuses to why an actual velocity is lower than its estimated is not meaningful. If you see an actual velocity that is lower than your estimated then you will want to lower your estimated velocity next sprint. “But”, you say, “the velocity needs to be higher, it is too low, the project will take too long”. … Yeah, right. But it is not the velocity that needs to be higher, it is the environment in which the team operates which needs to be different. You will only fool yourself (and the stakeholders) if you make up excuses and count on an optimistic, higher velocity than your actual; all you’ll manage to do is to make your plan look good, but it will then – inevitably – be a lousy, incorrect, optimistic and unrealistic plan. The one thing (except resources) that affects velocity is impediments. Remove impediments and the team’s velocity will go up. This should be your approach if you feel the velocity “is too low”.

Scrum Master role vs Leadership vs Project manager

2008-10-03T10:47:00.007+02:00

Is the Scrum Master "a leader"? Yes, an informal one - not a formal one. The Scrum Master should not perceive the role as a leader role. The Scrum Master should be someone the team trusts and looks up to - but should be someone within the team, on the same hierarchical level as the rest of the team members.

I'm pointing this out because I have noticed that being appointed "Scrum Master" may imply "becoming the team leader". Which is incorrect. I see the Scrum Master as a representative for the team - the "curling parent". A person sweeping a clean path for the team to go forward. That does not mean leading per se.

If the Scrum Master is "the leader" I think you risk losing some of the cooperation effect and the "tightness" of the team, and with a strong leading Scrum Master the team risks relying too much on that single person to take decisions and lead the way. Or at least it will be easy for the Scrum Master to fall back in old tracks and take charge: thereby undermining the self-organization and collective committment of the team.

So, when you appoint Scrum Masters, make sure it's someone within the team, who the team trusts, knows and looks up to. Most often you'll find there is one or a few informal leaders in the team, and either of those would be perfect for the Scrum Master role.

An important note here is that the Scrum Master should not be the tech lead of the team. Don't make the Expert the Scrum Master. Why? Because the Scrum Master's first and foremost priority will be to remove impediments and be "Scrum Police" (sorry, I mean "Advocate" :)). That means taking time away from other tasks, such as coding. So if you put your tech lead as Scrum Master you will lose pace and you will hinder the rest of the team (because the tech lead will be busy removing impediments and won't have as much time to help others).

Finally, a pitfall which we fell into - which I guess is not totally uncommon for organizations and people who are new to Scrum - is to mistake the Scrum Master role with the classic project manager role. When we started our Scrum implementation we thought that the project manager (called "Producer" in our world) should be the Scrum Master. That was indeed a mistake. Just read the above :-). A lesson learned here is that the project manager role is sortof altered and spread out across Scrum Master and Scrum Product Owner. In our implementation, we regard the Producer as the "Scrum Product Owner" nowdays. I'll write some more about that in future posts.

Don't document more than that you have to ask to understand

2008-10-02T15:51:00.006+02:00

Jeff Sutherland writes in his most recent blog entry about Agile Specifications. Interesting stuff!

I have rule of thumb that I use and try to teach everyone, when it comes to documentation: Don't document more than that you have to ask to understand. It is, I think, something to remember when it comes to writing agile specs, user stories, etc. Spending lots of time writing fancy specs and try to catch every little detail is just a waste of time, because I think it causes a false sense of trust in the spec. You won't succeed just because you discribe every little aspect of some feature. What will let you succeed is if the developers manage to understand it well enough to build it. And communication is key here. So don't write fancy specs just for the sake of it. Communicate instead! Ask! But as Jeff describes in his article above, of course you should document enough. We do write specs too (I'll describe our "scoping process" some day). But the above rule of thumb should always be kept in mind...

Picking stories at sprint planning = Drawing The Line

2008-10-02T15:13:00.007+02:00

My thoughts today is about how the team picks stories at the sprint planning meeting.

One point of the sprint planning is for the team to be allowed to pick the amount of work they feel they can cope with, and commit to, for the coming sprint. As a preparation for the sprint planning meeting, the product owner goes over the backlog and revises the story prioritizations based on most recent feedback/input from sprint reviews etc (whatever affects the priorities).

Our approach up until recently was to let the product owner present and make available (e.g. on a whiteboard) the X topmost stories in the backlog. Those stories would be considered "up for grabs"on the sprint planning, but without relative priority. The team could freely pick any of them, in any order. It sounded so good on paper; the team has complete freedom over which stories they pick, yet the product owner has a say about it by means of him prioritizing the backlog prior to the sprint planning meeting.

The pitfall with this is that since the team picks any stories from the top of the product backlog, the stories enter the sprint backlog without priority; or at least the priority has less of a meaning now. And it is therefore tempting to just regard the set of stories in the sprint backlog as one big unordered set of stories - all which must be completed. And as a consequence you lose one of the most valuable concepts of Scrum & Agile: focus on the most important thing first. Plus, it leads to people in the team isolating themselves with one story each, which means the cooperation effect is lost.

Don't do what we did!

So how should you do it? Well, the product owner should prioritize the stories in the backlog, and they should be ready-ready (see previous blog post). On the sprint planning meeting these stories are put in a list and the team can not just pick any of them. Instead, what the team does is to draw a line somewhere. That's it. This drawing of The Line represents their committment. Everything on one side of the line is what they commit to delivering, and the rest is left for consequtive sprints; or whatever the product owner choose to do with them. In our case, we should have the stories printed out on index card papers, and put on a whiteboard from left to right in descending priority order. The team then draws a line using a whiteboard marker. Simple enough! :-)

Ready-ready: the Definition of Ready for User Stories going into sprint planning

2008-10-01T11:30:00.008+02:00

You all know what “Definition of Done” means. Just for the hell of it I’ll recap it anyway; What it refers to is a state of a chunk of work. The state is also referred to as “Done-Done”. The purpose of the Done-Done mechanism in Scrum is to ensure that those things completed in the sprints really are completed in a commonly acceptable, controlled, premeditated and agreed sense. For example, nothing should be considered “completed” until it is e.g. tested, reviewed, committed and merged – e.g. “Releasable”. Without the definition of done we risk building up a mountain of stuff that we push ahead of us, which eventually (e.g. release-day) will kill us – or at least give us a hell of a fight. What’s even worse, without the definition of done we not only build up such a mountain of stuff, we also have no way of knowing how big that mountain is until we start getting our hands dirty and digging in. And what’s even worse is that many of the things inside that mountain will take so much longer to do later than if performed immediately day 0, because the longer it waits the harder it becomes to fix, to understand, to find, to remember, etc.

So, I think we’re all in agreement that we need a definition of done for the work we ship out of a sprint. Now I think we need a similar mechanism for the work taken in to the sprint. A mechanism which ensures that the stories taken in is really ready to be taken in. Let’s call this state “Ready-Ready”.

First of all, what does “taking in” mean? Well, it means “available and up for discussion on the sprint planning meeting”. So the definition of ready needs to describe a state of stories which they must be in, in order for them to be eligible for discussion on the sprint planning. Before they have reached the Ready-Ready state the stories do not exist and will not be considered for sprint planning. I.e. the top part of the product backlog has to contain only stories which are ready-ready.

Why? To save time of course: to spare the team tedious discussions about stuff that isn’t sufficiently thought-through. To let the team focus their valuable time and energy on things that are as “safe” as possible.

So, the definition of Ready should be;

A user story exists which points out the actor, describes the targeted feature and the purpose of it. The story should be formulated like this; As a I want to because . The first two are often easy to pinpoint but the purpose is often overlooked, because it feels obvious. But is it? The idea with writing down the purpose is to give the reader a good feel for the context of the feature – what problem it solves. Not only what it is but also why. This way, the reader can form his own opinion on how to implement it. It gives the reader a deeper understanding for the story.
The formulation of the user story is general enough for the team to have the flexibility of delivering it in increments. Don’t describe every detail. Describe roughly what you want but leave the rest for discussion.
The story is small enough to fit inside a sprint. Stories larger than that need to be broken down. Breaking down and reformulating stories have to be done before the sprint planning. Obviously, a story cannot be completed in a sprint if it is larger than a sprint. So care should be taken to ensure they are small enough.
Each story has its own unique priority in relation to every other story in the product backlog. This is to ensure that the team can work on the most important things first, and to force the product owner to make intelligent decisions about the order of stories.
The story has a description of how it can be tested (demoed). The description need to give the reader a good sense of what determines if the story is completed or not. It should be e.g. the acceptance test that is described here. It can be used as a description of how to demo the story too.
The story has been estimated (in story points). This should preferably be done by the whole or parts of the team, but at the very least it should be done by individuals with the same understanding of the system and the domain as the team who will eventually be working on it.

So in our case we should have meetings prior to the sprint planning meeting, where the product owner meets the teams and sits down and discuss the product backlog. Don’t forget to time-box the meeting. The goal of the meeting is to deem stories as ready-ready. It will require that some stories are broken down, reformulated, etc – and estimations are done using poker planning. A tester or test leader is required to attend here too, so that the formulation of the “how to demo” is really testable and meaningful.

Version control with multiple scrum teams

2008-09-30T16:00:00.010+02:00

Earlier this year Henrik Kniberg wrote a very interesting article about version control and scrum, in a multiteam environment. It doesn't cover every aspect, but it is a good source of inspiration and interesting to read anyway. You'll find it here: http://www.infoq.com/articles/agile-version-control

Cross-functional teams (“Feature teams”) in game development is not straightforward

2008-09-30T14:24:00.002+02:00

Our teams are average about 5-6 persons in size. Some are somewhat larger and some are smaller. But in general they are all a mix of people with varying competencies. For example for one of our games which is a somewhat “3D intense”, the teams consist of web developers (PHP), 3D developers (C++), 3D artists and 2D artists. The idea with Scrum is that the team collectively commit to the sprint backlog and help each other out to focus on finishing (Done-Done) stuff in sprint backlog priority order.

The problem here is that the 3D artists cannot really contribute to the PHP or C++ coding, and the Web developers aren’t really capable of helping with the 3D programming. So the feeling of collective commitment is weakened, and the cooperation-effect is lost; what remain to help out with are things like testing, perhaps code-reviewing and just generally “wiping the floor” (removing impediments) for whoever is working on the top-most priority stuff (e.g. being “Servants” to the “Kings”). It works sometimes, but it is not optimal. It would be interesting to know how others do it.

The alternative would be to split the teams up according to these “hard borders” of competence, so that e.g. the artists would be in a team on their own, and leaving all the programmers in one team. I don’t have empirical evidence to back this up, but I have a feeling it is a choice between two less-than-optimal alternatives, and putting them all into one team is less bad than to split them up…

Iterative, Incremental development - Continuous refactoring

2008-09-25T18:20:00.011+02:00

It's hard. Certainly much easier to say than to do, right? We keep failing at this. Actually, "failing" is maybe a bit harsh so let me rephrase; we're not as good as I think we should be.

The problem is in how we look at the code we write and the work we do; and what values and expectations we lay in the term "Done" (or "Releaseable"). We want to finish what we start. Even the Definition of Done (DoD) tells us that it should be "finished". Our Definition of Done says "Releaseable" (more or less). This should be your DoD too, by the way. So the team sits there with a bunch of stories to produce and knows that the DoD says that it should be "releasable". A natural reaction, given every developer's pride in work, is to focus on delivering something that is really fully "releasble", meaning "great". Or often "perfect". When all the DoD really (read it again if you don't believe me :-)) says is "releasable".

But here's the problem: "Finished" doesn't mean "Complete".

What we fail to do is to remember that the Definition of Done only requires us to deliver something that is possible to release. No more and no less. It doesnt say anything about the code being in super shape, future proof or completed-once-and-for-all-for-all-eternity-never-to-return-to-again. On the contrary; Scrum and Agile development is all about iterative, incremental work. It is about continous refactoring. It is OK, it is even expected - even required - for us to go back to code and add new stuff and kill off or change things done in previous sprints.

Forget about trying to do everything right from the start. Try and look only as far as the sprint length is - as far as your current sprint goal. That is what we need to focus on. We do what we need to now, in order to make what we're currently committed to - then we come back next sprint and add to and refactor what we previously did. It's ok - it is not something to avoid: it is something to strive for!

Another thing to consider is the following: isn't it great to be able to refactor your code, continuously? I know since my programming days, I really loved it when I got a chance to refactor my own code. This, I think, because I learned stuff when I originally wrote it - so I know how to make it better when I rewrite it. And I think most people love the feeling of making something better.

Finally, I also want to mention something I picked up when speaking to a scrum coach the other day: refactoring a few years ago really could be a pain, due to the poor development environments. But today most major IDE's and code-writing tools have great support for refactoring; in a matter of minutes you can make changes to tens or even hundreds of files. This is something that I will definately look into in our organization; what support for refactoring do we have in our development environment(s), and what better tools are out there?

Shock Therapy!

2008-09-22T15:36:00.004+02:00

I'm glad I'm not the only one with this experience... On the topic of what I wrote abot doing Scrum right from the start instead of cutting corners; I learned the other day that Jeff Sutherland (one of the "inventors" of Scrum) calls this approach "Shock Therapy". There's a very interesting article on his blog about this: http://jeffsutherland.com/scrum/2008/09/shock-therapy-bootstrapping.html

Estimating at Sprint Planning

2008-09-18T14:54:00.005+02:00

When do you estimate? And what?

We estimate stories using Gnome Days, which becomes the story's initial "Story Point" value. I know Scrum teaches to estimate the story points as a relative complexity or size, not taking time into account.

So you have a bunch of stories in your backlog, all estimated using Gnome Days. Meaning, I have knowledge about the size of the stories in the backlog, and with my knowledge about previous actual velocity I can derive an estimate of how long those stories will actually take.

So I go to the Sprint Planning meeting with a bunch of stories, the top priority stories in the backlog. The team picks a bunch that they feel comfortable with.

Now, how much do they look at the story point values when they pick what they commit to? Do they even care about it? At this point? Or do they adjust completely to it in relation to knowledge about past velocity? We do something in between. The team picks a set of stories they feel comfortable with but use the story point sum to compare against past velocity, as a sanity check.

Next question: Do you estimate the un-estimated stories at the sprint planning or are you supposed to have that ready when you enter? If so; at what point do you estimate stories? Does the Product Owner estimate the stories himself? Or is it done by the whole team at some other point (other than the sprint planning)? Or do you have an "estimation team" that meet regularly to estimate the most important currently unestimated stories in the backlog?

The latter is my preferred approach, until someone gives me a good argument as to why not. I like the idea of a relatively stable team of mixed competences who estimate stories ahead of time, in priority order. That way, that is done when the team enters the sprint planning, and the team can focus on picking stories and instead revise the estimates, breakdown into tasks where needed and/or sanity-check the commitment. How do other organization do it?

Start off by the book, if you're new to Scrum!

2008-09-17T21:34:00.008+02:00

We started our agile journey a year ago. I knew that Scrum was increasingly popular and I read a lot about it (and other Agile methods). One of the great things about it was (and is) that it is so "common sense". Every description of the method(s) explain that you have to find your own way and that for example Scrum is just a framework, not a ready out-of-the-box method or toolset. This is great, but it was also (to us at least) a pitfall and contributed largely to one of the first really big mistakes (out of many ;-))...

Why? What happened? Oh, right.. I will explain. The pragmatic attitude of Scrum and the "laidback" sortof informal tone it gives when you read about it, made us feel like it was alright to cut corners; "Oh, yeah well we dont have to do it exactly by the book - Agile and Scrum says you need to find your own way right, an implementation and process set that suits you and your unique environment.. So then it's OK for us to skip the retrospections really, or to not have people appointed Scrum Master' [and so on]". We'd tend to use Scrum and Agile and its flexibility and humbleness, if you will, as an excuse to cut corners in the implementation. As a result, I think our implementation of Scrum/Agile has been suffering and consequently has taken a lot longer time than it otherwise could have. Don't get me wrong; the Agile approach has been meaningful and has improved our means of developing software from day one. I just feel it could have done us even more good even sooner. Many times throughout the past year we've discovered something new about Scrum that we've introduced (or had a problem that we've had to solve) which has really made us take one or several steps closer to a "real" Scrum implementation.

What I am saying is this: If you're new to Scrum and you want to introduce it in your organization or your team; do it by the book first. Don't cut corners or make tradeoffs! Set out to follow everything you read about it, 100%! Take that extra week of studying Agile and Scrum, or that extra $2000 CSM course. Do that, upfront from day one. No seriously. Do it.

I wish we had done that. You will want to learn and fully understand the method first and the thinking behind it, before you are capable of understanding how to make tradeoffs or cut corners. Yes, Scrum is a very flexible method (framework) and you definately need to adjust to your organizaiton in order for it to work fully for you. But you need to (really) understand it first before you're capable of understanding the full implications of shortcuts.

You will find a valuable checklist that you want to benchmark against, here, it's Henrik Kniberg's Scrum Checklist. I wish we had known about this sooner. Check it out.

Backlog manager tool

2008-09-16T20:04:00.009+02:00

Just wanted to share our "Backlog Manager" excel file, which we use in all our teams.
I'll try to make some posts here with further explanations, information and thoughts surrounding it, but for now I'll just settle on providing the link to the file.

It should work both for Microsoft Excel 2003 as well as Microsoft Excel 2007. It is possible to open in OpenOffice Calc too, but I don't think the Macros work.

NOTE! Buttons doesn't work (nothing happens when you click them?)... Well then you have to enable Macros in the security settings. See the little notice that appears just below the menu/toolbar in Excel when you open it.

Here's the link:
BacklogManager-v8.xls
(Updated 2009-01-23)

Credit to Henrik Kniberg for the original, which I have incorporated into this sheet. The index card generator and the index card template is full credit to him (although I refined his version a little and edited parts of his VB code).

Tool for recording Planning Poker games

2008-09-16T18:11:00.008+02:00

When we play Planning Poker I wanted a way of recording the results of each story in a simple way. I also wanted to be able to accept the fact that we won't always reach consensus. So if we have three people playing a 3, person playing a 2 and one person playing a 5, and all of them refuse to change their minds, what do you do? Do you count it as 3? Or 5? Or 2? After you've tried letting the extremes argue (defend) their choices and then had a couple of replays, it starts to feel silly. So being a fan of mathematics and formulas I wanted to be able to live with that difference in opinion and still have a clear way of counting the result. The obvious approach would be to count the average; (2+3+3+3+5)/5 = 3,2 (or rounded to nearest half; 3.0). It works. Do it if you want. I on the other hand wanted the formula to tilt towards "worst case" rather than letting the lower estimates affect it equally. So I used a formula I picked up in (I think) "Agile Estmiating and Planning" by Mike Cohn; Estimate = (1*Min + 2*Avg + 3*Max)/6. The result in the example above would be: (1*2+2*3.2+3*5)/6=3.9. Or rounded to nearest half (which we always do); 4.0.

So for every play we do we record the final estimates from each player. We enter them in the formula above and that is what we the put as the estimate or story point value.

I have a tool I made in Excel, which we now use. Feel free to download and use if you like:
EstimationSheetTemplate-v4.xls

Planning Poker and tasks vs stories

2008-09-16T17:37:00.005+02:00

We've used Planning Poker for over a year now. Our experience is that it is a great and easy way of coming up with estimates. If you don't know what it is, you can read more about it here.

When we started it seemed natural to us then that the values of the Planning Poker deck represented hours; 1,2,3,5,8 hours and so on. We had a rule of thumb that said that no task/requirement/story should be larger than e.g. 18-24 hours. If estimators felt like playing a higher card than that then it indicated that the story needed to be split up into smaller bits.

So, this seemed fine, especially on paper. The problem was that many sessions became extremely lengthy and we often got stuck in discussions about details and solutions. I know this is to be avoided in Planning Poker, but that's what I wanted to point out here: we made that mistake. It wasn't as easy as just saying "No let's not talk about design details now"... We said that. Over and over :-). People felt they needed to discuss the details, in order to decide between e.g. a 5 or an 8, or between 8 and 13... It was often possible to force a play and ask people to follow gut-feeling and intuition first, but the discussions kept popping up over and over again.

Our approach on avoiding this and speeding up the sessions was to simplify: forget hours. We estimate days instead. Our smallest estimate now is 0.5 (days) and then we follow the typical Planning Poker deck series; 1,2,3,5,8 and so on. A typical task/story takes between 0.5 and 5 days, I would say.

In my experience estimating in hours causes a false sense of security and leads to over-confidence in the plans. One of the arguments against using days as the estimation unit may be that we then miss the opportunity of really thinking through what parts/activities/tasks a certain story/task consists of and thereby miss possibly important aspects in the estimate... Maybe it is so. I dont know yet... So far I think it's a good tradeoff; I trade waste of time and false sense of security for a possible decreased accuracy... Well, I've already accepted to sacrifice accuracy given the 2-3 minutes per story and the whole idea behind Poker Planning to begin with: a fast and simlpe way of estimating.

My understanding of the common practice in Scrum is that you estimate two sorts of things;
1. Estimate stories in Story Points, which is a relative estimate of complexity (regardless of time)
2. Break stories down into tasks and estimate tasks in hours.

Our implementation (so far :-)) is to simply estimate stories using Gnome Days. Sometimes we have to break stories down into tasks during a sprint planning session, but certainly not always. If we do, we obviously estimate the tasks individually, but we still use days as the unit, not hours.

I'm curious to hear about other people's experience in estimating days vs hours in Scrum.

Gnome Days

2008-09-15T18:32:00.001+02:00

So how does other people estimate?

I know, estimate relative size or complexity. That's all good. But how do you do it? In my experience it's not really natural. There's always a discussion first about neglecting (or the impossibility of neglecting) the time factor in the estimation process.

The idea with relative complexity estimation is - as I've understood it - to ignore time and instead focus on (in lack of a better word) the size of the task/feature/requirement/story. This means that you should ignore thinking in hours or days and instead just order the stories relative to eachother; this is twice as hard as this and this is about the same, and this is half as hard, and so on. Henrik Kniberg's approach of Turbo Estimation is perhaps the simplest form of complexity estimation; you just divide each story into simple, medium and hard.

This is all good. My problem with the above mentioned method is that it doesnt "feel" natural.
Every time I've initiated an estimation session with the intent of estimating complexity acc to the above, it has sparked (sometimes) lengthy discussions about why it works or not.

My usual argument is that the estimators should think of it as time being a result of the complexity and the term complexity should be regarded more like "size". E.g. the time is a result of the "size" of the work of completing a certain task/requirement/story/feature.

Another complication here is the lack of continuity in the estimation process. As you know, measuring velocity (actual vs estimated) is a key point in Scrum in order to manage sprint intake, so you will want the complexity estimates to mean the same thing from one estimation session to another in order for velocity to actually have a meaning. And since we usually dont have estimation sessions more than maybe once every two or three weeks people tend to forget the relative size of stories across sessions.

My conclusion from the above is that sure, relative estimation of complexity is great in the absence of any other method, but it really leaves many things to be desired.

Our solution now is to estimate in ideal days, but we call it "Gnome Days" - or Tomtedagar in Swedish. The term was inspired by Henrik Knibergs book "Scrum and XP from the trenches" (if you haven't read it yet: do! It's free and can be found here, although I suggest you buy it to support Henrik) where he describes that time estimation should be done considering how long a task will take to complete for an optimal number of resources locked in an room and left completely undisturbed. I thought of that as a gnome (swedish "Tomte") in a basement, who you lock up and don't let out again until the task/story/requirement is completed.
Anyway, we estimate the initial "story point" value of each story using gnome days. This is in our experience much more natural and still lets estimators estimate stories relative to eachother. A challenge here is to avoid micromanagement and microestimation, so we forbid the use of hours. Instead, our smallest estimate (ever) is 0.5 gnome day. If a story is estimated to more than e.g. 3-4 gnome days then it could probably use breakdown into smaller stories.

Scrum ftw!

2008-09-15T14:18:00.005+02:00

Yes. Scrum ftw! Actually, Agile ftw! As I study the methods more closely I become increasingly convinced of the strengths - which lay in the simplicity. In the common sense.
I just started this blog today so there's not much here yet, but I plan on describing ideas and sharing tools and lessons here, from the journey that I've had the chance of being a part of during the past year or so, in this company.