Updated backlog manager tool in Excel

I have updated the Backlog Manager tool now, it's a bit simplified compared to last time. Changes since previous version:

- It works in Microsoft Excel 2003 now (previously only worked in 2007).
- You may now freely delete and insert rows in the backlog without risking that the list gets screwed up (due to missing formulas and such).
- There are no hidden columns or weird formulas in the Excel.
- The Card Generator no longer uses the ID-column, but instead the Autofilter result, so you can now filter with full flexibility and the Card Generator will generate cards for the visible rows only.
- The New Story dialog is simplied and looks better :-)
- The index card template looks better and is somewhat simplified.
- Some redundant columns were removed (simplify ftw! :-))

I have updated the old post from September last year so it links to the new version of the Backlog Manager tool / Excel instead.

Anyway, you can find it here too:
BacklogManager-v8.xls

Good luck!

User Stories from the trenches

I couldn’t resist the topic. But for the record, credits to Henrik Kniberg for the “…from the trenches”-bit :-).

The past few weeks have been hectic, and I haven’t really had the time to post anything new in a while. The reason is that I’ve been in a phase here where we’re ramping up a project and I have been heavily involved in trying to form an initial backlog with an initial set of reasonably meaningful User Stories for that project. I.e.: we’ve been involved in getting a (very large) project “vision” down into a set of stories.

Our traditional method for this was to have the Product Manager (Customer) produce a written description of each feature that he wanted, and then he would hand that over to the Development Department where we (“we” being the team and the Scrum Product Owner: as you may know we use the somewhat controversial “Product Owner Proxy” construction in our organization) would take ownership of that written “spec” and break it down into User Stories that were prioritized and later estimated and delivered in priority order.

My problem with that approach was (and is) that the Product Manager usually (for whatever reason) spent alot of time writing these “functionality specs”.
I have felt that this goes against my core attitude/rule-of-thumb “Never document more than that you have to ask to understand” and that it is pretty "un-scrum". I’ve been wanting to find a better and more Agile way of moving from “The vision inside the head of the Product Manager” to “Something that the team understands, which can be prioritized and brought into the Sprint Planning meetings”.

In this case, with this development project being such a large one (a new product, with so many functions), it would simply have been an overwhelming task for the Product Manager to rely on our traditioinal approach and write all those "function specs" up front – at least considering the usual level of detail in those spec documents.

My idea was to skip the “detail specification” step alltogether and jump straight from the vision to User Stories, and thereby save time by having the Product Manager produce User Stories directly which later could be prioritized and further broken down. This would somehow force the Product Manager to skip details and thus not try to think of everything up front. Oh, and of course, we’d be forced to rely much more on face-to-face communication between the team and the Product Manager, which would be great: “two people by a whiteboard” is always a many times more efficient method of transferring information than by document, right?

So that was the plan, and this is what I have been struggling with for the past weeks (plus celebrating Christmas and New Year :-)).

I can’t say that it went great maybe, but I'm still pretty satisfied with the result so far. Anyway, I wanted to take this opportunity to really reflect over how things went and try to gather my own thoughts about it and nail down some lessons.

Here are some of the challenges with the past few weeks;

• I needed to find a way for the Product Manager to “empty his head” into something that could be used as basis for formulating some fundamental stories.
• I needed the Product Manager to understand what User Stories are and why they should be formulated in a certain way, what level of detail they should have, and so on.
• I needed to find a way – a method and tool – to document the User Stories which would fit nicely with the tools we use further down the road: we use the Backlog Manager tool that I have made available for download on this blog in a separate post.

I approached this by introducing MindJet MindManager to the Product Manager. I know there are open source and other free alternatives out there too, but I haven’t tried any of them myself.

I use MindManager for many different things, and I find it great for emptying my head down into symbols, boxes, notes and relations. So the idea was that it could be used by the Product Manager too, to get the rough set of functions and other thoughts and notes down into a visible map. That map could hopefully then be used to look at to formulate User Stories; e.g. by studying a subtree and all the nodes there to remember what needs to be covered by the backlog.

The map ended up looking something like this, but much bigger (the map below is just a very small dummy example, obviously):

In retrospect, we experienced a few major drawbacks with this method of using a mindmap to describe the vision of the project and have the Product Manager use it as basis for formulating stories;

1. Coping with keeping the map up-to-date with the stories: as stories are formulated, iterated, thought about, discussed and reformulated, after a while there are changes in the perception of how things are supposed to work. We failed at keeping the map up-to-date throughout this process. After a while the list of user stories conveyed a different picture than the map. However, I think it might be ok that the map is a “consumable” that can’t be trusted after a while; it’s just one tool used along the way and is not really an artifact by itself.
2. Ones stories were formulated, they ended up being too small. The initial backlog ended up containing much too many stories and each story gave just a too fragmented view of what should be delivered. The symptom of this was that it was very hard for the team to grasp the project as a whole, even with a lot of communication with the PM. The breakdown done was simply much too detailed and too early.

I think my main lesson here is to try to get the Product Manager to just summarize each desired feature as a Theme or Epic (see separate blog post), rather than in a whole bunch of small stories. If there are important details about the function that the Product Manager wants to mention he should write keywords as “notes“ to the story, and not try to write each as a separate story.

Also if there are things in the Epic/Theme which can be given different priorities, obviously those can be broken out as separate Themes/Epics. The ambition should however be to not break everything down in detail.
The finer breakdown, into “ready-ready” (see separate blog post) User Stories should be done by the team in cooperation with the Product Manager – not by the Product Manager himself.

So to summarize on the lessons learned so far:

• Get the Product Manager to use a mindmap to empty his head into, with regard to a certain vision of function(s).
• Get the Product Manager to look at the map and its subtrees and use that to write Themes (or Epics), formulated as User Stories (“As a I want to so that .”.
• Get the Product Manager to write those Themes/Epics in the same or similar tool that you will be working with in Development (e.g. if you can, directly into an empty Backlog Manager excel, if that’s the format you’re using).
• When the Product Manager feels he’s covered the most obvious and/or most important Themes/Epics, bring in the team. Let the team and the Product Manager together work on the finer breakdown – of the most important Themes/Epics.
• Talk alot about the Themes/Epics: between Product Manager and team. Let the Product Manager describe each Theme, if you have an updated mindmap you can use that as basis for a presentation about each, as part of the breakdown process.
  
• Break down the Themes/Epics one by one, and work face-to-face. Even if it takes the PM and the team a long time, I have a feeling that it is time saved in the end: because the team will get an excellent view of the backlog. A challenge here is to heavily moderate discussions about how to build the stuff: solutions shouldn’t be discussed at this point. That’s done when the stories enter the sprints.
• Don’t attempt to break every Theme/Epic down into smaller stories from day one. Make sure they’re explained and moved into the product backlog, but break them down in priority order: start with the most important ones. Leave the lower priority Themes/Epics for later sessions. Just because they’re big it doesn’t mean that you can’t estimate them. Remember: rather roughly right than precisely wrong.
• Come back to the bachy hklog after every sprint and review the Themes/Epics and let the team and the PM break them down once they surface in the backlog.

Here is an example of a subtree which could be used as basis for writing a Theme and later for discussing around as part of breaking it down into smaller stories;

Example of a Theme for the above subtree: "As a Administrator I want to be able to manage users so that I can control who has access to what."

Once the team and the Product Manager break it down together, they'd probably end up with something like the following user stories:

• As an Administrator I want to be able to add new users to the system so that each user can have his own personal account to login with.
  
• As an Administrator I want to be able to search for users and and be able to sort the resulting list in various ways so that it maximally easy to find what I'm looking for.
• As an Administrator I want to be able to Clone an existing user so that I create a new one with the identical data pre-filled so that I don't have to retype everything.
• ...and so on.

We’re still not done here and have some distance to go before we can say that the backlog is in an “OK” shape. I’ll revise this post and/or make additional posts with more thoughts and experiences as we go along.

Oh, by he way. I always appreciate feedback about your experiences. If you have any clever ideas or thoughts about this or how you work with it in your organization, please leave me some comments.

User Stories, Themes and Epics

In a coming (much longer) post about story formulation, I refer to the terms “Epic” and “Theme” and I just wanted to clarify what that means to me.
I figured I might as well make it a separate post since it’s somewhat a topic of its own.

• A “User Story” is a backlog item. nothing strange there, its everything you think it is. Its format is “As a I want to so that .”.
• A Theme (to me at least) is a large User Story.
• An Epic (to me at least) is a VERY large User Story.

For example, if you’re building “user administration features” your typical User Story, once it enters a sprint, might be something like “As an Administrator I want to be able to add users so that I can give a person an account to login with.”. That could be the “normal” size of a story that is far up in the product backlog, i.e. close to development. It’s pretty small.

Earlier in the process (further down in the backlog) it could be represented as “As an Administrator I want to be able to administer users of the system so that I can determine who has what access to the system.”. Notice the difference. This is a Theme. It’s much bigger and much less detailed than the story above. But it’s still not massive, and with some discussion it’s possible to figure out roughly what reasonably is hiding behind that formulation.

Earlier still (even further down in the backlog), the formulation could have been something like “As a System Owner I want various administration features so that I can control the behavior of the system.”. This story could be considered an Epic because it is so large and so generic and loosely formulated that it can cover a huge amount of different things.

Kings and Servants - collaborate on the highest priority item

Some time ago I heard a great suggestion on how to view the idea of a team “swarming” on the top-priority item of the sprint backlog – i.e. how the team should see themselves with regard to who works on what story. As you know, following the priority order is a key mechanism in Scrum affecting your ability for timeboxing (having a flexible scope).

The tip was to view the people who work on the topmost priority item in the sprint backlog as “Kings”. Everyone else (consequently working on lower priority items) are Servants. All the Servants help the King(s) immediately as soon as the Kings need anything. Also, everyone wants to be King.

I’m working on spreading this idea in our organization. I had barely finished speaking to one of our teams before I found a printed picture of a crown put up on the Scrum Board alongside the top priority story :-).

Here's an image that I found with Google that you can print and put magnetic tape on the back of and use on your own Scrumboard(s).

Done or Complete? How to develop in iterations.

This topic is closely related to that of “Continuous Refactoring” which I am constantly nagging about. It is also related to “Definition of Done”, which is another favorite topic of mine :-).

When are you “Done”? The right answer is: When the Story meets the definition of done. Let’s say your Definition of Done says “Releasable”. So, the Story is done when it is Releasable. Great. But, wait… What does that mean, “Releasable”?

Coders are creative people. I touched this subject in my previous post “1-day remaining forever”. We want to do an excellent job, not just an average one. We want everything we ship to be just right, because bad solutions or sloppy code reflects directly on us and our abilities and creativity. So it’s easy to get stuck in the “almost done” mode, coz we want to polish that extra method or just fix that last design flaw or just rewrite that last class which was poorly written two years ago... So when faced with the goal line (i.e. the point when we’re “Done”) we really want to go to lengths to make sure everything we’ve done is really “ready” and fully thought through, and future-proof, and so on.

But the point I am trying to get to here is that there is a difference between “Done” and “Complete”. Every story needs to be Done in the sense that it fulfills our Definition of Done. But that does not mean “Complete”: it does not mean e.g. “future-proof” or “with architectural perfection”. Those states are achieved over several stories – over several iterations. This is why we call Agile development Iterative. So, think of it like this: “Done” is something we do with each story and each sprint, and “Complete” is something we reach over several stories and several sprints.

I’ve seen it several times; developers dig down into the system and try to solve problems that are of much less priority than the story at hand, e.g. by designing or preparing for some story that is much further down in the product backlog. This behavior builds dependency between stories, it breaks prioritization and it adds a lot of insecurity to the estimates. The proper approach is to focus on the immediate story only, and effectively staying within its boundaries. Trying to stay within the boundaries of a story requires discipline and (not least) a good understanding of Scrum and continuous refactoring.

To sum things up: Aim for “Done”, not “Complete”.

Until next time..!

Flexible scope - A magic bullet?

Everyone agrees that you can’t both have a rigid deadline and a rigid scope. We’ve been there, Done that, Got the g***amn t-shirt. Software Development is not mass production and last time I checked I wasn’t able to foretell the future. We want to avoid projects where both the scope and the date are fixed. That’s one of the first lessons every developer learns… And that is one of the key things that Scrum brings: you develop in iterations, you timebox and you ensure you have something to deliver all the time, especially release day. Yay!

Timeboxing means the date should be regarded as fixed. Wow, there’s an easy sell to upper management…! But as a consequence we have to allow the scope to flex… Simple enough? Well, allowing the scope to flex might not be as easy as it seems. “Flexible scope” as a concept is easy enough to understand, but it’s not something that happens by itself just because you say “we do Scrum”, develop in iterations with beautiful sprint planning sessions and sprint reviews.

I’ll illustrate with a scenario that I’ve seen in reality several times:

Let’s say you’ve started using Scrum, and one of the key selling points to upper management was: “In Scrum we never move the deadline, we promise to try to always have something releasable at the deadline. I can’t promise exactly what, but we won’t ever need to move the date”. Good start!

So, now you start working. The team is working on a release of (let’s say) two major features (each consisting of several smaller “parts”). The Scrum Product Owner and others work on breaking down the scope(s) into User Stories. But they’re not really familiar with Scrum and the stories they produce are poorly formulated, and are done without them really understanding what a User Story is and what to think of.

Still, eventually a product backlog is ready. But, again since the Scrum Product Owner might not be that familiar with Scrum there is not much effort or thought put into the prioritizations of the stories in the backlog. Anyway, the team starts their sprint with a sprint planning meeting and grabs whatever is on top of the backlog.

Once rolling, the team execute the sprint with little or no regard to the priorities. Everyone grabs a story each and start working.

Work continues along those same lines sprint after sprint. And stuff actually do get done. Progress is OK, people feel OK. So far so good?

Now, when you approach the intended release date, you look closer into what’s been done so far and you conclude that there are still several critical things that aren’t ready yet, or that maybe haven’t even been started yet. In addition, you conclude that there are critical tasks remaining for each of those stories which you previously thought were “Done” – tasks such as integration testing, string translation, code reviewing or similar. Ouch.

At this point, what are your options?

1. The "easy" way out is to postpone the release date, i.e. effectively breaking the timebox rule. …Scrum breaking down. Oh misery…! And you had such an easy sell to upper management with the fixed-date thing... And all you do is move the date and then have the same problems once more, but that time you've sortof used up your goodwill and no longer have the luxury (?) of being able to postpone the date...
2. Crunch! There's that word again... But you think "Just this time… This release was special...". You make up excuses about why you’re where you are, and call for overtime. Have people come in a little earlier and stay a little later. Have them come in those last few weekends. Not to mention those last days before deadline. So, the team works overtime, they put in extra hours, and not surprisingly quality suffers, people suffer, shortcuts are taken everywhere… Anyone recognize this?
3. Flex the scope by lifting out the entire feature from the release. Since there are still critical stuff remaining you don’t have the option to just abandon parts. All you can do is lift out the whole feature. That doesn’t give such a good impression of the whole “flexible scope” mechanism, now does it? “Yeah well, in Scrum we don’t move the date, so we have to let the scope change, and that means removing a feature from the release. We have to allow that. That’s Scrum.”. A job well done. No, this is not what “flexible scope” means.

Those are really the only three options. And what’s even worse is that this is probably not discovered until very late. With only days or a few weeks (best case?) remaining until the scheduled release date… This is Scrum failing big-time!

So, the point I want to make here is that we need to be very careful so that we really maximize scope flexibility. The main mechanisms in Scrum which are affecting this are:

1. Strict prioritization
2. Ability of adhering to Definition of Done == Releasable.
3. Skillfulness of breaking down a scope into user stories

To me “strict” means that we both prioritize AND stick to it, by actually working on things in priority order. If we don’t have a prioritization we can’t really say that we have done all in our power to have everything “critical” or “most important” (or whatever value you want to put in the prio) ready by the time the timebox ends. If you’re not careful, the “completed” part of the scope may be any combination of important and less important things, and (what’s worse) the same is true for the stuff that isn’t done yet.

Prioritizing the backlog needs to be intelligently done, and only 1 person should be allowed to have the final decision about a story’s priority: the Scrum Product Owner. I recommend thinking of the importance in terms of “Business Value”. But be careful; it doesn’t only mean value as in dollars and cents (or Swedish Krona and Öre ;-)). The keyword here is Business. Think of it in terms as Value to your Business, including your own internal business of developing stuff. So things that for example are “architecturally significant” (I’ll post an article separately about this topic, some day) might get a high Business Value because it is important for the architecture and the consecutive development of other stories - even though the story itself may have less of an obvious value to the end user, the customer, your accounting department or your shareholders (but remember: if it is architecturally significant it is most likely valueable to the customer and end user too, it's probably just a matter of thinking about it hard enough and being able of formulating it in words).

The Definition of Done applies to how you “complete” stories. I’ve posted entries here in the past about this and I won’t go into details here. I just want to point out that adhering to the Definition of Done affects your ability of flexing the scope, because you ensure that you stay in control over what’s remaining; you stay on top. The things that have been done are really “done” and are really releasable, there are no hidden piles of work under the rug. I hate surprises and Definition of Done is my best friend :-).

A largely contributing factor both to your ability of adhering to the Definition of Done, and to your ability of following a prioritization, is how good you are at breaking the scope down into User Stories. Don’t regard the scope as a big monolithic lump of functions (e.g. the waterfallish term “Requirements”). Instead, you need the stories to be independent, finishable, valuable and estimatable. A good guide on the way of thinking you want to have is the I.N.V.E.S.T. acronym (stories should be Independent, Negotiable, Valuable, Estimatable, Small and Testable). Writing stories is a skill that you have to work actively with and continuously try and improve.
Don’t be alarmed though; it’s not rocket science. Some breakdown is better than none, and you shouldn’t make such a big deal out of it. Learn by doing. Try different approaches to formulating stories, try the INVEST acronym, try other ways. As long as you understand that the “quality of the breakdown” will affect a lot.

So, to wrap things up: with intelligent scope breakdowns, by following the prioritization and the Definition of Done, we can be more than reasonably sure that the e.g. 85% of the scope which is “ready” when the timebox ends, is really ready, and the remaining 15% is not half-started and we don’t have dependencies between the ready 85% and the remaining 15%. We know we can release something – and that something is actually something valuable (hopefully even the maximum value).

There. Long post this time. But I like this subject. And it's an important one. I’d be glad to learn about other peoples’ experiences. Cheers! My food is getting cold.... ;-)

Have you ever looked at a burndown, really?

The topic today is about burndown charts. I’d like to make some remarks about what it is used for and why we have it.

First of all, the Burndown Chart is a pretty obvious artifact of Scrum. If you use a wall or whiteboard as your “Scrum Board”, you probably have the burndown close-by and you probably (you should) update it daily. As with most artifacts and mechanisms of Scrum, the basics of the burndown chart is pretty easy to understand. So, at a glance, the burndown chart is self-explanatory.

But is it?

Introduction to Burndown Charts
Let’s start out with discussing what the burndown chart is, mathematically/practically; The burndown is the sum of remaining time for all those items in the sprint backlog, spread out over time along the sprint. So along the X-axis you have your timeline, typically measured in calendar days (at least work-days). Along the Y-axis you have remaining time. You can measure that in Days, in Hours, in Gunships or whatever you want. I recommend that you measure it in (people-)days, though. So, on the daily scrum (the standup meeting) the team goes through all the items in the sprint backlog and they figure out the current remaining time (e.g. by summing up all tasks, or whatever level of detail you have on your scrum board). The sum is plotted in the burndown chart. The result is a plotted line that (hopefully) moves from the top left corner, to the lower right corner. By observing this line the team and everyone else can clearly see how well “on track” the team is: will we reach 0 by the time the sprint is ended?

Can the burndown be "ugly"?
Another important remark (that maybe and unfortunately isn't so obvious) is that the burndown chart is intended as a tool for spotting problems early (by displaying progress daily), which means that it has to be reliable: it has to be the real progress, the hard cold truth - not some made-up half-truthful data that makes the chart "look good". Ever thought of a burndown as "ugly"? I know I have. Sorry, but it is not the burnchart that is ugly (don't kill the messenger): it's the reality. It's the context in which the sprint is executed that is "ugly". As soon as you start "dressing up" the burndown you start hiding the truth. And who will end up biting the dirt for it in the end? Yes, the team...

So, a precondition for having any use what so ever of the burndown chart is that it is as truthful as it is current/up-to-date.

A deeper use of Burndown Charts
But why do you do this at all, track progress and plot it in a burndown? Why display it in this way? Or more specifically: what do you (you, the team, the scrum master, everyone) do with this information?

I know for my own part that I’ve observed hundreds of burndowns by now, and so have all of my colleagues. But I’m not sure that everyone has really spent time reflecting over what the burndown charts mean, other than the immediate, most obvious: that we can see if we have more work remaining than what will be reasonable to finish by the end of the sprint. So, that’s why I wanted to bring this topic up: to get a chance to reflect over how to use the burndown in a deeper, more analytical way: more effectively.

Also, I think most people just reflect over the single, immediate burndown infront of them, without specifically observing or thinking about the burndown in the context of other burndowns from past sprints.

The burndown, in combination with the Scrum Board is an excellent and powerful tool that tells a lot about the immediate climate and environment of the team and the ongoing sprint. If you have knowledge about burndowns and velocity of past sprints, then you have even more powerful information. The burndown – and a set of several burndowns from past sprints – is like a flashlight revealing problems. As soon as there is a problem with progress it is shown in the burndown. But equally important: if you have problems with planning, with product backlog maintenance, with outside disturbance or even with your Scrum implementation as a whole, it will show in those burndowns – if you just know what to look for, and if you realize that this is possible.

So, I guess my advice is: take time and reflect not only over the current burndown, but also reflect over the last 2-3-7-whatever sprints and their burndown, at the same time. I suggest you take a photo of the sprint burndown chart at the end of each sprint and archive it together with records of the estimated and the actual velocity. Then bring that out now and then, let the team get their hands on it and spur discussions about how it looks. Discuss patterns and findings.


Scenarios to reflect over
Below are a couple of made-up burndown charts that I made in Visio (yay!) to visualize some key points. I intended them to represent two consecutive sprints each.

Two sprints showing an interesting tendency...

What would it mean if you see a pattern like above? Here, already from the first few days of each sprint, it is possible to see where the burndown is headed. It is however left like that for too long time, but at least the team acts at some points, by (probably) removing stories from the sprint. So that’s good. But the next sprint shows the same tendency. What if this happens a third time, and a forth? It is clear that the team is over-committing every sprint. I.e. they are not following their actual velocity carefully enough. They are ignoring reality (velocity) and convincing themselves that “next sprint it will be different” – but it never is. So, by observing the burndown over a few sprints you can see that (1) the team acts by removing stories (great), but (2) the team keeps over-committing. It is shown in the burndown as well as in the actual compared to estimated velocity.

Two sprints that show flatlines...

What about the pattern above? Well, the team is unable to complete items – for some reason. It is not resolved (or new problems with the same result keeps appearing) over several sprints. So, the obvious issue here is that the impediment perseveres and renders the team unable to finish items. The not-so-obvious problem here is that nobody seems to act to resolve it. Two sprints is a long time. Remember that Scrum says that impediments should (optimally)be removed in a day. So you really have to issues here that you need to resolve: (1) fix the impediment stopping the team from finishing items, and (2) find out if and why nobody is addressing that impediment.

The two examples above are easy ones, but still I’ve seen them both. I think a lot can be saved by observing both the current burndown and really reflecting over it, but also (not least) by now and then observing the burndown of several sprints in a row.

Pair programming according to Crisp

Read a very interesting article on Pair Programming today, written by Jan Grape of Crisp. Sorry it's only in Swedish, but here you go.

Allow slack, don't allocate 100%

A classic approach to resource allocation is to aim for 100% allocation. Anything else would be cost-inefficient, right? Having people sitting around doing nothing can’t be a good thing, right? …Well, wrong…!

Aiming at 100% allocation would be desired only if it was possible to know and plan every detail that every person has to do, in advance. And you can’t. Allocating less than 100% doesn’t mean that the unallocated time is expected to be spent on youtube (no offence, youtube is great! ;-)). No, it means that the unallocated time is expected to be available for all unplanned things that has to be done in order for everyone to do their planned work and keep the pace up.

It’s important so let’s repeat it: You should allocate less than 100%. The remainder, let’s call it Slack, is expected to be available to do useful things that can’t be planned for. What are those “useful things”? Well, helping your teammates for example. If you don’t allow for slack the team members won’t have time to help each other. So the team’s productivity will actually decrease because the cooperation effect is lost.

Usually the purpose of trying to allocate to a 100% is to make sure that productivity is maximized. But, in fact increasing allocation towards 100% will risk giving the opposite effect.

Obviously, the trick is to have just enough slack.

So, slack is good. You want slack, and you need to stay weary of any attempts or implied desires to the decrease slack time.

There are several methods you can use to ensure you introduce the slack time as a mechanism in your planning:

• Continuously measure and use “Focus Factor”. A typical focus factor might be somewhere between 0.6 and 0.8 I would say, but there are no rules and differs for every team. So don’t focus too much on this number: it’s a crude measurement and it has much more to do with how large/many impediments and how much disturbance the team has, how good your user stories are and how good your backlog is, than e.g. the team’s efficiency.
• Expect no more than e.g. 6 hours of "available work-time" every day. We’ve been experimenting with values as low as 5.5 and up to 6.5 or 7. If you use a project tracking tool it should be possible to set “Working hours” somewhere as a global setting for your project. Either as a percentage of a working day, or as a start- and end-time of each day (which was the option in our case).
• Measure actual velocity and compare to actual, continuously and let the team help out to give feedback on their sense of “load”

We’ve tried all three, and I think a combination of them is the way to go, depending on situation. Key is to understand why you need slack. If everyone understands that, then you’re able to use either method whenever it feels right.

For another, shorter and more pedagogical way of saying what I say above, I recommend Henrik Kniberg’s slide that you can find here (see page/slide number 26): http://old.crisp.se/henrik.kniberg/presentations/2008-10-13-HiQ/Introduction-To-Lean.pdf

Fixed price contracts: Money for nothing, Change for free

Just a tip: If you haven't seen or heard of it already, read this recent blog post by Jeff Sutherland about the concept of "Money for nothing, Change for free" which is his name on the approach of setting up development contracts; Agile Contracts.

I haven't applied this approach myself, but it is really a fascinating topic. I come from the consulting business so I've experienced first-hand the horrible reality of Waterfall classic fixed price contracts. I'd be very interested in trying to apply Agile and Scrum to such a situation...

Bug-fixing inside inside sprints or outside?

Today’s post is about how to manage bugs.

First some background. We’re a product developing company and we maintain our own products as they go live. A lot of our work focus on new feature releases for the existing products. We have continuously updated lists of live bugs that has to be dealt with.
Even if you’re not a product developing company like we are, I’m sure that you can relate to the situation.

Ok. Let’s take a minute and look at what a “bug” is, and its difference to “a story”. A bug is something that is often unestimateable. Trying to estimate it may take longer than actually fixing it. Or estimating it may even mean fixing it. A “bug report” is a description of an unwanted behavior of the system or, not seldom, a question about a possible unwanted behavior. Anyway, hopefully it also has some type of “priority” relative to other bug reports (and hopefully there’s a single person who prioritizes – or you’ll have a mess). Of course, bugs are unwanted and we need to strive towards continuously improving so that we avoid bugs altogether. The day we have zero bugs I’ll erase this blog post ;-). But until then we need a way to deal with them.

Since recently we use Trac to manage our bug lists. We used to use Hansoft. We moved to Trac because we felt it was more suited for our needs, and it has an excellent and highly configurable web interface. It is also possible to integrate to Subversion, which is the version control system we use.

Now to the options: Do you bring those bugs into the sprint backlog during sprint planning? Or do you leave room in the sprint backlog to be able handle bugs during the sprint, and maybe even set aside time and resources for it? Or do you have “Bug-fixing Fridays”? Or do you have dedicated teams for it? Or maybe you have “Bug-fixing Sprints”? The options are many, and we of course haven’t tried all. I’ll try here to summarize some of the most obvious methods that I can think of though;


A team of dedicated resources work in parallel to the development teams
This has been the approach in one of our products, for almost a year now. We have two dedicated full-time developers who work with bugs. It works great but it requires these resources to be highly multi-talented. These guys don’t work in sprints (at least not with the bug fixing) and they set aside time for the bug fixing activities; on a few specific days of the week they work with bug fixing and release patches to the live product. The other days these guys work with other “urgent” things which we want to shield from the development teams – much of these things have to do with supporting other internal functions such as sales, advertising, marketing, customer support, etc. We call this team “Client Services Team”.

You’ll need to be careful so that these guys are kept happy. If they want to circulate into development you should try to accommodate for it. Working with bugs can be tiring and stressful. Another downside to this is that the developers who originally cause the bugs seldom get to fix or even see the problems so the bugs doesn’t really serve as an incentive for them to continuously improve. On the upside though the release cycles are short and the bug fixing guys often see immediate result of their work. Also, they get to see and learn to work with a huge variety of technologies and parts of the system.


Dedicate an amount of time during each sprint for bug-fixing
This is our current approach in another one of our products, for a couple of months now. We (most often) do 1-week sprints, so the cycles are short anyway. Not sure this approach would work if we had longer sprints. But we let the development team assume the responsibility for dealing with bugs. They do this by means of two mechanisms;

• Bugs as stories: every week the Scrum Product Owner and the Scrum Masters meet to discuss what’s currently critical or high urgency in Trac, and appropriate issues are taken out and put into the product backlog as stories. They attempt to (guess)estimate them using the StoryPoint value (Gnome Days, see other blog posts below). These stories are merged into the backlog just like any other story, and the prioritization is made relative to any other stories in there.
• Dedicated time every week for “other” bugs: every week (decided on the sprint planning) the team appoints a person and a specific day of the week which will be used for bug fixing (bugs that are not already in the sprint as stories). On that day, Trac is checked for the current buglist and bugs are fixed from top to bottom. Bugs here are not fixed in order of priority but rather in order of size (smallest first). “Smaller bugs” get done faster, regardless of their priority. The idea is to avoid having bugs in Trac open for ages which are really annoyances but that are not critical. I think maybe this sorting on size instead of priority is a sign of the prioritization mechanism not working properly…

This approach seem to work well too, but it requires that the Scrum Product Owner and Scrum Masters really take time continuously to look at the buglist and update the product backlog properly.


Dedicate a whole sprint every now and then
We haven’t tried this approach. I think there are some problems with it. For one, the team will lose pace and we’ll break the Agile principle of “maintaining a constant pace indefinitely” (see http://www.agilemanifesto.org/principles.html). Secondly, the cycle time from bug report to fix will most likely be longer. This approach doesn’t work well for critical bugs that has to be fixed “right away”, unless the sprints are ultra short. Those critical things would most likely end up leaking in to the development sprints. The upside could be that the developers would get to fix their own bugs. It would also force a wider spread of competence throughout the development team(s) instead of e.g. limiting it to certain individuals.


Bug-Fixing Fridays
We haven’t tried this approach either. It sounds interesting enough though. I'm curious to find out if anyone's tried it. It would mean that you regularly dedicate a certain day(s) for everyone to do bug fixing. Maybe every week, maybe more seldom. The problem with this approach though is that it may be hard to have a feeling of continuity in the bug fixing process. And bugs that take longer than a day to fix will end up open for a long time.

On IDG's article about critisism against Scrum

Here's a Swedish article from IDG about that "the critisism against Scrum is growing".

It has something of an eye-catching headline, but the article is otherwise thin. The only really valuable thing in there are the statements from Tobias Fors about having to understand Scrum before trying to adjust it. But the topic as such and the angle of the article is just rediculous. Also, I think Tobias was somewhat misquoted or misunderstand with regard to what is written in the article about retrospectives.

The author quotes Ivar Jacobsson about the teamsize and iteration length limits of Scrum being a downside, as opposed to other methods. The problem with that argumentation is that Scrum is then blamed for not allowing "large teams" or "long iterations". You have to stop and ask yourself why. Why do Scrum limit the team size and the iteration length? Obviously because statistics and years of experience in software development projects has shown that those are factors directly affecting the outcome of a project.

That Scrum doesnt suit "large systems that are based on a service oriented architecture or large organizations" as a general statement is gong too far. If Scrum works or not is in my opinion much more a matter of the organization's willingness and capability to try - not a problem with Scrum itself.

Sure, some of the advantages of Scrum are lost if you are strictly controlled by a ruleset that is waterfall-based, such as public fixed price tenders, or if you work against the FDA or similar. In those cases the problem isn't with Scrum. The problem is with the very preconditions for the projects. In that sense this Ivar Jacobsson really do has a point. But for the author to write an article under the topic "critisism against Scrum" on that is to me a sign of news drought.

Traditional roles vs Scrum roles

Ok, you have an "ordinary" organization with traditional roles, and you want to do Scrum.
You have someone with “Project Manager” or “Producer” written on their business card. Maybe you have someone who’s “Technical Project Manager” too. You have your team, obviously. And last but not least you have some Customer. In our company we do our own products, so we don’t have external customers in that sense. We do have users who are extremely important of course, but I regard them as end-users rather than assigner in the sense of who’s ordering work from us in production. In our case the assigner is the person with "Product Manager" written on the business card – she’s the one making the business calls and the one outmost responsible for the direction and the result of the product. In the world of consulting the customer is obviously the external party who orders and pays for the work done. Anyway, the team does the work. The Project Manager is usually responsible for the high and the low; from high-level planning and risk management down to resource allocation, problem-solving, task assignment & follow-up, etc. The customer makes the order and expects delivery. Below you’ll see a schematic of the traditional roles.

Ok, so we’ve ironed out how things usually look. The above would apply to organizations with external customers (like consulting companies) as well as organizations with internal customers (like us, who own and distribute our own products) – it’s just a question if the Customer is an external or internal role.

Now, one simple approach when introducing Scrum is to put the Scrum Master-hat (SM) on the project manager. He’s the one doing many of those duties anyway, like helping the team move forward, keeping track on who’s doing what, the one controlling the status meetings, etc. Let’s say you do that, so then what roles are left to figure out? Yeah, the Scrum Product Owner. Ok, that sounds simple enough, at least in a company like ours where we already have someone internally who’s “owning the product” and making the business decisions: the Product Manager becomes the “Scrum Product Owner” (SPO hat). Even the name of the Scrum role suggests this construction. Now, in a consulting company, with external customers, I’m not sure it’s as easy as that. I’ll leave that open for now. Anyway, here’s how it would look:



Wow. You now have a team, the person who used to be “project manager” who is now called “Scrum Master” instead, and the person who used to be Product Manager is now called Scrum Product Owner. Sounds alright, right?

Wrong! But what’s wrong with this picture?

A lot of things. One of the most obvious things with this setup is that the Scrum Master isn’t someone within the team. The ever-so-important purpose and work of the Scrum Master is missing or at least the efficiency will be lower because of it. See previous blog post below about why the Scrum Master shouldn’t be a formal leader and needs to be someone within the team. Another factor here, if you have some history in your organization, you’ll need to remember that the project manager has been the one putting the strain on the team, the person who’s been breathing down the team members’ necks. The one with the whip. So it’s a person that the team will most likely never see as “being on their level”; a fact which will make it hard to perform the duties of the Scrum Master to any meaningful extent. So what can you do to change that? Maybe what you want to do is give the Scrum Master hat to someone in the team? Voila, you have your roles. And the project manager is out of a job. Or?



But wait, so far we haven’t discussed the appropriateness of giving the product manager the SPO hat. We just assumed it’s best because the names of the roles are similar :-). I argue that you need to carefully consider whether or not the product manager should really be your PO. In fact, I think that instead the project manager is very suited to take on that role. Because what happens is that the “project manager duties” are smeared out across (A) the team and the Scrum Master, and (B) the SPO. The daily stuff, the things that have to do with who does what, when and why, is handled by the team and the Scrum Master, and the planning, prioritization, risk management and so in is done by the PO. And so, the product manager in this scenario (or in the case of a consulting company – the customer) is perfect for being regarded simply as “Customer”, nothing else.

Now, I know there is some people arguing out there about whether or not “Product Owner proxies” should be used. Here's a link to a Swedish article by Tobias Fors on that subject. As I understand it, the argument from those opposing it is; if the person with the outmost responsibility (e.g. the assigner – the customer) for the product is not deeply involved with the team and instead those duties are taken over by the former project manager – now called the PO – then the PO becomes just a “proxy” for the real product owner. And a key point in Scrum is to involve the stakeholders early, with the team, and get the feedback directly and quickly. Hence, relying on a proxy for it will negatively affect that. So the argument is to forget about proxies and instead let the real product owner be the one with the Scrum Product Owner hat. Well, I have to agree with that. But there’s a reality too, where the product manager is loaded with business decisions, meetings, research, spreadsheets, budgets, administration, etc. In an ideal world I would change all that and remove all those things from the product manager’s back so she could focus on the important parts; what goes in the product – and thereby give the Scrum Product Owner role to her. But we don’t live in an ideal world and I will have to make do with what I can get. And the next best thing is a really engaged project manager (who I call “Scrum Product Owner”), who has the direct link to the product manager (who I call “Customer”).

So, finally, my recommended setup is as follows;


With this setup, some of the advantages are (not in any particular order);

• The Customer/ProductManager can focus on other things too, and does not have to spend all his/her time on helping the team get the job done. He/she still has to be around on a regular basis and provide feedback early, at least on or around every sprint review.
• The Scrum Product Owner is a full-time role who's job it is to (a) know the customer and (b) know the team. The challenge here lay (among other things ;-)) in how well this person understands what the customer wants, and how well that understanding is reflected in the prioritization of the product backlog and the breakdown into stories.
• The Scrum Master is someone within the team, not a formal leader. Also, with this construction the Scrum Master can really focus 100% on being the one sweeping a clean path for the team ("the curling parent"), without any specific further responsibilities outside those of the SM and those of the whole team as a collective.

In an ideal world, if I practically had the choice (e.g. if I were to build up a new organization from scratch and got the power to decide every aspect of it) then I would try and let the "Product Manager/Customer" have the SPO hat. In that sense I do agree with the concerns about "Proxy Product Owners". But if that is not possible - which I don't see that it is in our case, and probably in many other cases - then I chose the pragmatic approach. I think the above is the better than... Well, what's the alternative? To have a Scrum Product Owner or a Scrum Master that for various reasons aren't focusing 100% on their roles.

"1 day remaining" forever

Let’s say the team is working on some story and need to finish a task which is originally estimated to 4 days. Once the team starts working they continuously decrease the estimate until they reach 1. The task then remains on “1 day remaining” for several days – even though the team actually do work (and hard, too) on finishing the task. You ask the team what’s going on, why the task never reach 0, and the reply you get is “Well, things keep popping up that we didn’t think of which has to be done too in order to finish this task. We can’t finish the task unless we also handle those things.”. Let’s say you don’t only see this occasionally. You see it over and over again, Story after Story and Sprint after Sprint.

What is the problem? (Please leave me some comments below if you’ve seen this problem). I have some likely suspects:

Estimation technique
I.e. your estimation process. If you keep having this problem then you probably need to consider if it is because you keep missing something in the estimation. Maybe it is always the same “thing” that keeps holding the task up from being finished? Could you incorporate that “thing” as an item in a checklist which is always considered during estimation meetings? Do you have the right people attending the estimation meetings? Do you have someone with deep enough, and right, knowledge about the system and technical domain to give his/her input to the estimate?

Technical Debt
As a system ages, the shape of its code, architecture, design etc deteriorates – its natural, and every coder knows it. The problem is that the rate of decay is so much higher if the people working on the system continuously take shortcuts. There’s a saying that I keep coming back to; “The dirty remains long after the quick is forgotten”. Think of that “dirt” as taking a loan. Eventually you will have to pay it back, and in the meantime you will have to pay interest on the remainder. “Interest” in our world is in the form of us having to deal with problems when developing new things; things inevitably take longer because the system is so complex. Paying back the loan will mean having to take time to eliminate those shortcuts (that dirt).

If your team tends to stay on “1 day remaining” for ages then you’ll need to consider if the problems they have (the unknown stuff that appears) are due to technical debt. If it is, you’ll want to prioritize very high some efforts for paying back some loans. There’s a business reality out there so, most often is simply not possible to halt development for six months and have all resources work only on refactoring. But at least you can ask yourself; are we increasing the debt still as we build new stuff? And if so, why? Most likely the developers do not want to take shortcuts, but they are more or less forced to because of pressure (implicit or explicit) from stakeholders. You’ll want to look at your team’s environment and see if you can somehow remove the feeling of them not being allowed to take longer time to do things “right”.

A mechanism to emphasize this is to introduce a bullet to your Definition-of-Done: “Not increased technical debt”. It should mean that it is not OK for a task/story to be finished until the implementation is such that it doesn’t increase the technical debt. I.e. it WILL inevitably take longer to finish, but the shape of the code will be better than otherwise. Next step is to replace that bullet with the following: “Decreased technical debt”. That will mean that with every story that is finished there has been some effort put in to refactor (even if ever so slightly) what’s already there.

Formulation of User Stories
If your stories are formulated in a bad way then the team will definitely have a hard time finishing. Maybe the stories are too big? Maybe there’s too much detail and no room for flexibility? If your stories contain every little detail then you’re really back to waterfall again. You’ll spend enormous amounts of time writing specifications, leaving no room for the developers to think for themselves and no way for them to build it in any other way than what’s specified. And hence, it will be naturally hard to develop in iterations. The problem is that the flexibility of the scope has been lost. You essentially end up with a fixed scope.

“Fixating a scope”, to me, is OK. Wow, that’s a controversial statement. Let me try and explain what I mean. As long as it is done on Theme/Epic level (i.e. something along the lines of what’s traditionally called “Function Specification” level) – and not on Story level (i.e. not like a Requirement Specification), then it is OK for it to be considered fixed. If you have someone (e.g. a customer) ordering something (e.g. a feature or a whole system) then it’s OK for that person to really think through what he/she expects. It’s even OK if that is written down.

But as the feature(s) come closer to the development team then it is important that it (however detailed) is broken down into User Stories which are flexible enough to be implemented in iterations, so that you are able to deliver something working at the end of every sprint.

If you’re the Scrum Product Owner, then spend some time thinking about how the stories are formulated. Is the level of detail “just right”? Is there enough flexibility in there for the team to be able to finish it (i.e. without being extremely restricted to an exact certain scope)?

Badly overlapping User Stories
Obviously there’s a million ways to derive (Mike Cohn calls it “trawl”, by the way) User Stories from a vision of a feature or system. I’ve stated previously that I believe the approach should be to divide the feature(s) into verticals (we call them “slices”). Regardless of if you listen to my advice or not :-), you risk running into the problem with overlapping stories. Note that I do not mean “overlapping” as in overlapping code/subsystem: on the contrary, continuous refactoring is all about iterating and reiterating the same code over and over again. What I mean is overlapping functionality; the problem comes when you can’t finish a Story because it intertwines with other Stories, in a way that you can’t tell which story you’re really working on. That is a badly overlapping User Story.

If the team keeps remaining on 1 day, then you should consider if maybe badly overlapping stories is contributing. Try reformulating stories with this in mind. Can you think of other ways to write the stories? Think of the acronym “INVEST” as a rule of thumb; stories should be

• Independent
• Negotiable
• Valuable
• Estimateable
• Small
• Testable

Finished does not mean Complete
This is, I think, as much a result of all of the above as it is a fundamental attitude in the people working on and around the system. When a Story is “Finished” it doesn’t mean that what has been done is in any way “Complete”. I keep coming back to it: Continuous refactoring, iterative development – this is what it is all about.

When unforeseen problems appear and a team stay on “1 day remaining” for ages, can it be that they put the wrong meaning in the word “Finished”?

“Finished” should mean that the story is implemented in some way. It doesn’t mean that the implementation has to be perfect, and it doesn’t mean it needs to be completed and never again returned to. I’ll post a separate blog some day about whether or not this means increasing technical debt or not (in my opinion it does not).

This, I think, requires a change in attitude among the developers, the product owner and the customer/stakeholders. A sprint demo/review doesn’t mean showing Complete stuff. It means showing Stories finished. “Finished” as in “Done-Done”, yes. But that doesn’t mean that we don’t go back immediately next sprint and add, change or remove stuff in order to finish another Story.

Also, another related cause may be that the sprint is too long. The longer the sprint, the more and the heavier the feedback will be from stakeholders. The more seldom the stakeholders see the result of a sprint the more they will feel that they have to “get all the comments out there” and thereby the bigger deal the sprint review will be to the team. And the bigger deal the more it implies “Complete”. So, consider shortening the sprint length (a lot) and get the customer and other stakeholders involved much more often. It is a matter of building confidence between team and customer – confidence that the result in the end is what was expected (or actually even better).

Beyond the buzzwords

What is “Agile”? And what is “Scrum”? And “XP”? Oh.. And Lean? What is that? Can you “work with” Agile? Or what exactly is it that you do? Are all just different and c00l buzzwords meaning the same thing?

Oh, the confusion… =)

Well, I think of Agile as a “value package”. The word “Agile” is simply the name of a set of core values and principles about how “development projects” is viewed. So, you can’t really “work with Agile”, but you can have processes and project methodologies that are Agile – they fit the Agile core values. Here’s what’s written on Wikipedia about Agile Software Development: “Agile methodologies generally promote: A project management process that encourages frequent inspection and adaptation; a leadership philosophy that encourages team work, self-organization and accountability; a set of engineering best practices that allow for rapid delivery of high-quality software; and a business approach that aligns development with customer needs and company goals.”. There’s a manifesto that covers the core values, which you can find here: http://agilemanifesto.org/.

So, “Scrum” then, what is that? Is that a process? Or a is it also just a “value package”? No, to both questions. Scrum is an agile "process framework". It is not a process, it is a framework for processes to exist (be formed) within. Scrum is an interpretation of Agile Development. Right or wrong, I view Scrum as an “instance” of an Agile development approach. Scrum sets the boundaries and the general direction for the processes but it doesn’t state exactly what those processes should be or what steps and methods they need to hold. That is why it is said that if you “implement scrum” you have to find your own way: you have to create your own set of processes and practices that fits your organization and the Scrum framework. It is quite correct to say that you “do Scrum”: it means that you have identified and enforced a set of practices that cover all the aspects of Scrum. You can read more about Scrum on (lots of places) for example http://www.scrumalliance.com/ and on Wikipedia.

Ok, and “XP” (eXtreme Programming)? Just like Scrum, XP is an “instance” of an Agile development approach. So XP too needs interpretation and adaptation to be implemented in your organization; it is not an out-of-the-box ready set of processes. The difference between Scrum and XP is that Scrum is more focused on project management and team aspects of software development, while XP focuses almost entirely on the hands-on development practices. Scrum for example talks about how to plan your iterations, using Sprint Planning meetings and Sprint Reviews. It doesn’t say much about how to actually write code. XP on the other hand doesn’t say as much about managing your project (although there is some overlap) but focuses much more on how the developers need to sit together and work in pairs, or that you need to have continuous integration, automatic testing, Test Driven Development (TDD), and so on. Scrum and XP fits really nicely together, and together they form a great and whole-covering framework for a really agile development approach. In our organization we have yet to adopt XP to any extent. But we’re getting there.
You can read more about XP on (lots of places) for example Wikipedia.

And finally, what is “Lean” in all of this? Well, it too is a way of observing the world - the world of manufacturing to be precise. According to Wikipedia, Lean is “the practice of a theory of production that considers the expenditure of resources for any means other than the creation of value for the presumed customer to be wasteful, and thus a target for elimination.”. It originates from the automotive industry (e.g. Toyota in the 1950s) and was later adapted to software development (Lean Software Development) by Mary and Tom Poppendieck. Just like Agile it covers a set of core values, and sets a direction for how to go about “manufacturing”, in our case Software Development. Lean is all about spotting and removing sources of “waste”, decreasing cycle time, continuous improvement, respect for and personal growth of individuals, etc. Lean and Agile fits very nicely together.

Confused? Yeah, me too ;-)