Thursday, October 30, 2008

Allow slack, don't allocate 100%

A classic approach to resource allocation is to aim for 100% allocation. Anything else would be cost-inefficient, right? Having people sitting around doing nothing can’t be a good thing, right? …Well, wrong…!

Aiming at 100% allocation would be desired only if it was possible to know and plan every detail that every person has to do, in advance. And you can’t. Allocating less than 100% doesn’t mean that the unallocated time is expected to be spent on youtube (no offence, youtube is great! ;-)). No, it means that the unallocated time is expected to be available for all unplanned things that has to be done in order for everyone to do their planned work and keep the pace up.

It’s important so let’s repeat it: You should allocate less than 100%. The remainder, let’s call it Slack, is expected to be available to do useful things that can’t be planned for. What are those “useful things”? Well, helping your teammates for example. If you don’t allow for slack the team members won’t have time to help each other. So the team’s productivity will actually decrease because the cooperation effect is lost.

Usually the purpose of trying to allocate to a 100% is to make sure that productivity is maximized. But, in fact increasing allocation towards 100% will risk giving the opposite effect.

Obviously, the trick is to have just enough slack.

So, slack is good. You want slack, and you need to stay weary of any attempts or implied desires to the decrease slack time.

There are several methods you can use to ensure you introduce the slack time as a mechanism in your planning:
  • Continuously measure and use “Focus Factor”. A typical focus factor might be somewhere between 0.6 and 0.8 I would say, but there are no rules and differs for every team. So don’t focus too much on this number: it’s a crude measurement and it has much more to do with how large/many impediments and how much disturbance the team has, how good your user stories are and how good your backlog is, than e.g. the team’s efficiency.
  • Expect no more than e.g. 6 hours of "available work-time" every day. We’ve been experimenting with values as low as 5.5 and up to 6.5 or 7. If you use a project tracking tool it should be possible to set “Working hours” somewhere as a global setting for your project. Either as a percentage of a working day, or as a start- and end-time of each day (which was the option in our case).
  • Measure actual velocity and compare to actual, continuously and let the team help out to give feedback on their sense of “load”
We’ve tried all three, and I think a combination of them is the way to go, depending on situation. Key is to understand why you need slack. If everyone understands that, then you’re able to use either method whenever it feels right.

For another, shorter and more pedagogical way of saying what I say above, I recommend Henrik Kniberg’s slide that you can find here (see page/slide number 26): http://old.crisp.se/henrik.kniberg/presentations/2008-10-13-HiQ/Introduction-To-Lean.pdf

Tuesday, October 28, 2008

Fixed price contracts: Money for nothing, Change for free

Just a tip: If you haven't seen or heard of it already, read this recent blog post by Jeff Sutherland about the concept of "Money for nothing, Change for free" which is his name on the approach of setting up development contracts; Agile Contracts.

I haven't applied this approach myself, but it is really a fascinating topic. I come from the consulting business so I've experienced first-hand the horrible reality of Waterfall classic fixed price contracts. I'd be very interested in trying to apply Agile and Scrum to such a situation...

Thursday, October 23, 2008

Bug-fixing inside inside sprints or outside?

Today’s post is about how to manage bugs.

First some background. We’re a product developing company and we maintain our own products as they go live. A lot of our work focus on new feature releases for the existing products. We have continuously updated lists of live bugs that has to be dealt with.
Even if you’re not a product developing company like we are, I’m sure that you can relate to the situation.

Ok. Let’s take a minute and look at what a “bug” is, and its difference to “a story”. A bug is something that is often unestimateable. Trying to estimate it may take longer than actually fixing it. Or estimating it may even mean fixing it. A “bug report” is a description of an unwanted behavior of the system or, not seldom, a question about a possible unwanted behavior. Anyway, hopefully it also has some type of “priority” relative to other bug reports (and hopefully there’s a single person who prioritizes – or you’ll have a mess). Of course, bugs are unwanted and we need to strive towards continuously improving so that we avoid bugs altogether. The day we have zero bugs I’ll erase this blog post ;-). But until then we need a way to deal with them.

Since recently we use Trac to manage our bug lists. We used to use Hansoft. We moved to Trac because we felt it was more suited for our needs, and it has an excellent and highly configurable web interface. It is also possible to integrate to Subversion, which is the version control system we use.

Now to the options: Do you bring those bugs into the sprint backlog during sprint planning? Or do you leave room in the sprint backlog to be able handle bugs during the sprint, and maybe even set aside time and resources for it? Or do you have “Bug-fixing Fridays”? Or do you have dedicated teams for it? Or maybe you have “Bug-fixing Sprints”? The options are many, and we of course haven’t tried all. I’ll try here to summarize some of the most obvious methods that I can think of though;


A team of dedicated resources work in parallel to the development teams
This has been the approach in one of our products, for almost a year now. We have two dedicated full-time developers who work with bugs. It works great but it requires these resources to be highly multi-talented. These guys don’t work in sprints (at least not with the bug fixing) and they set aside time for the bug fixing activities; on a few specific days of the week they work with bug fixing and release patches to the live product. The other days these guys work with other “urgent” things which we want to shield from the development teams – much of these things have to do with supporting other internal functions such as sales, advertising, marketing, customer support, etc. We call this team “Client Services Team”.

You’ll need to be careful so that these guys are kept happy. If they want to circulate into development you should try to accommodate for it. Working with bugs can be tiring and stressful. Another downside to this is that the developers who originally cause the bugs seldom get to fix or even see the problems so the bugs doesn’t really serve as an incentive for them to continuously improve. On the upside though the release cycles are short and the bug fixing guys often see immediate result of their work. Also, they get to see and learn to work with a huge variety of technologies and parts of the system.


Dedicate an amount of time during each sprint for bug-fixing
This is our current approach in another one of our products, for a couple of months now. We (most often) do 1-week sprints, so the cycles are short anyway. Not sure this approach would work if we had longer sprints. But we let the development team assume the responsibility for dealing with bugs. They do this by means of two mechanisms;
  • Bugs as stories: every week the Scrum Product Owner and the Scrum Masters meet to discuss what’s currently critical or high urgency in Trac, and appropriate issues are taken out and put into the product backlog as stories. They attempt to (guess)estimate them using the StoryPoint value (Gnome Days, see other blog posts below). These stories are merged into the backlog just like any other story, and the prioritization is made relative to any other stories in there.
  • Dedicated time every week for “other” bugs: every week (decided on the sprint planning) the team appoints a person and a specific day of the week which will be used for bug fixing (bugs that are not already in the sprint as stories). On that day, Trac is checked for the current buglist and bugs are fixed from top to bottom. Bugs here are not fixed in order of priority but rather in order of size (smallest first). “Smaller bugs” get done faster, regardless of their priority. The idea is to avoid having bugs in Trac open for ages which are really annoyances but that are not critical. I think maybe this sorting on size instead of priority is a sign of the prioritization mechanism not working properly…
This approach seem to work well too, but it requires that the Scrum Product Owner and Scrum Masters really take time continuously to look at the buglist and update the product backlog properly.


Dedicate a whole sprint every now and then
We haven’t tried this approach. I think there are some problems with it. For one, the team will lose pace and we’ll break the Agile principle of “maintaining a constant pace indefinitely” (see http://www.agilemanifesto.org/principles.html). Secondly, the cycle time from bug report to fix will most likely be longer. This approach doesn’t work well for critical bugs that has to be fixed “right away”, unless the sprints are ultra short. Those critical things would most likely end up leaking in to the development sprints. The upside could be that the developers would get to fix their own bugs. It would also force a wider spread of competence throughout the development team(s) instead of e.g. limiting it to certain individuals.


Bug-Fixing Fridays
We haven’t tried this approach either. It sounds interesting enough though. I'm curious to find out if anyone's tried it. It would mean that you regularly dedicate a certain day(s) for everyone to do bug fixing. Maybe every week, maybe more seldom. The problem with this approach though is that it may be hard to have a feeling of continuity in the bug fixing process. And bugs that take longer than a day to fix will end up open for a long time.

Wednesday, October 22, 2008

On IDG's article about critisism against Scrum

Here's a Swedish article from IDG about that "the critisism against Scrum is growing".

It has something of an eye-catching headline, but the article is otherwise thin. The only really valuable thing in there are the statements from Tobias Fors about having to understand Scrum before trying to adjust it. But the topic as such and the angle of the article is just rediculous. Also, I think Tobias was somewhat misquoted or misunderstand with regard to what is written in the article about retrospectives.

The author quotes Ivar Jacobsson about the teamsize and iteration length limits of Scrum being a downside, as opposed to other methods. The problem with that argumentation is that Scrum is then blamed for not allowing "large teams" or "long iterations". You have to stop and ask yourself why. Why do Scrum limit the team size and the iteration length? Obviously because statistics and years of experience in software development projects has shown that those are factors directly affecting the outcome of a project.

That Scrum doesnt suit "large systems that are based on a service oriented architecture or large organizations" as a general statement is gong too far. If Scrum works or not is in my opinion much more a matter of the organization's willingness and capability to try - not a problem with Scrum itself.

Sure, some of the advantages of Scrum are lost if you are strictly controlled by a ruleset that is waterfall-based, such as public fixed price tenders, or if you work against the FDA or similar. In those cases the problem isn't with Scrum. The problem is with the very preconditions for the projects. In that sense this Ivar Jacobsson really do has a point. But for the author to write an article under the topic "critisism against Scrum" on that is to me a sign of news drought.

Monday, October 20, 2008

Traditional roles vs Scrum roles

Ok, you have an "ordinary" organization with traditional roles, and you want to do Scrum.
You have someone with “Project Manager” or “Producer” written on their business card. Maybe you have someone who’s “Technical Project Manager” too. You have your team, obviously. And last but not least you have some Customer. In our company we do our own products, so we don’t have external customers in that sense. We do have users who are extremely important of course, but I regard them as end-users rather than assigner in the sense of who’s ordering work from us in production. In our case the assigner is the person with "Product Manager" written on the business card – she’s the one making the business calls and the one outmost responsible for the direction and the result of the product. In the world of consulting the customer is obviously the external party who orders and pays for the work done. Anyway, the team does the work. The Project Manager is usually responsible for the high and the low; from high-level planning and risk management down to resource allocation, problem-solving, task assignment & follow-up, etc. The customer makes the order and expects delivery. Below you’ll see a schematic of the traditional roles.


Ok, so we’ve ironed out how things usually look. The above would apply to organizations with external customers (like consulting companies) as well as organizations with internal customers (like us, who own and distribute our own products) – it’s just a question if the Customer is an external or internal role.

Now, one simple approach when introducing Scrum is to put the Scrum Master-hat (SM) on the project manager. He’s the one doing many of those duties anyway, like helping the team move forward, keeping track on who’s doing what, the one controlling the status meetings, etc. Let’s say you do that, so then what roles are left to figure out? Yeah, the Scrum Product Owner. Ok, that sounds simple enough, at least in a company like ours where we already have someone internally who’s “owning the product” and making the business decisions: the Product Manager becomes the “Scrum Product Owner” (SPO hat). Even the name of the Scrum role suggests this construction. Now, in a consulting company, with external customers, I’m not sure it’s as easy as that. I’ll leave that open for now. Anyway, here’s how it would look:



Wow. You now have a team, the person who used to be “project manager” who is now called “Scrum Master” instead, and the person who used to be Product Manager is now called Scrum Product Owner. Sounds alright, right?

Wrong! But what’s wrong with this picture?

A lot of things. One of the most obvious things with this setup is that the Scrum Master isn’t someone within the team. The ever-so-important purpose and work of the Scrum Master is missing or at least the efficiency will be lower because of it. See previous blog post below about why the Scrum Master shouldn’t be a formal leader and needs to be someone within the team. Another factor here, if you have some history in your organization, you’ll need to remember that the project manager has been the one putting the strain on the team, the person who’s been breathing down the team members’ necks. The one with the whip. So it’s a person that the team will most likely never see as “being on their level”; a fact which will make it hard to perform the duties of the Scrum Master to any meaningful extent. So what can you do to change that? Maybe what you want to do is give the Scrum Master hat to someone in the team? Voila, you have your roles. And the project manager is out of a job. Or?



But wait, so far we haven’t discussed the appropriateness of giving the product manager the SPO hat. We just assumed it’s best because the names of the roles are similar :-). I argue that you need to carefully consider whether or not the product manager should really be your PO. In fact, I think that instead the project manager is very suited to take on that role. Because what happens is that the “project manager duties” are smeared out across (A) the team and the Scrum Master, and (B) the SPO. The daily stuff, the things that have to do with who does what, when and why, is handled by the team and the Scrum Master, and the planning, prioritization, risk management and so in is done by the PO. And so, the product manager in this scenario (or in the case of a consulting company – the customer) is perfect for being regarded simply as “Customer”, nothing else.

Now, I know there is some people arguing out there about whether or not “Product Owner proxies” should be used. Here's a link to a Swedish article by Tobias Fors on that subject. As I understand it, the argument from those opposing it is; if the person with the outmost responsibility (e.g. the assigner – the customer) for the product is not deeply involved with the team and instead those duties are taken over by the former project manager – now called the PO – then the PO becomes just a “proxy” for the real product owner. And a key point in Scrum is to involve the stakeholders early, with the team, and get the feedback directly and quickly. Hence, relying on a proxy for it will negatively affect that. So the argument is to forget about proxies and instead let the real product owner be the one with the Scrum Product Owner hat. Well, I have to agree with that. But there’s a reality too, where the product manager is loaded with business decisions, meetings, research, spreadsheets, budgets, administration, etc. In an ideal world I would change all that and remove all those things from the product manager’s back so she could focus on the important parts; what goes in the product – and thereby give the Scrum Product Owner role to her. But we don’t live in an ideal world and I will have to make do with what I can get. And the next best thing is a really engaged project manager (who I call “Scrum Product Owner”), who has the direct link to the product manager (who I call “Customer”).

So, finally, my recommended setup is as follows;


With this setup, some of the advantages are (not in any particular order);
  • The Customer/ProductManager can focus on other things too, and does not have to spend all his/her time on helping the team get the job done. He/she still has to be around on a regular basis and provide feedback early, at least on or around every sprint review.
  • The Scrum Product Owner is a full-time role who's job it is to (a) know the customer and (b) know the team. The challenge here lay (among other things ;-)) in how well this person understands what the customer wants, and how well that understanding is reflected in the prioritization of the product backlog and the breakdown into stories.
  • The Scrum Master is someone within the team, not a formal leader. Also, with this construction the Scrum Master can really focus 100% on being the one sweeping a clean path for the team ("the curling parent"), without any specific further responsibilities outside those of the SM and those of the whole team as a collective.
In an ideal world, if I practically had the choice (e.g. if I were to build up a new organization from scratch and got the power to decide every aspect of it) then I would try and let the "Product Manager/Customer" have the SPO hat. In that sense I do agree with the concerns about "Proxy Product Owners". But if that is not possible - which I don't see that it is in our case, and probably in many other cases - then I chose the pragmatic approach. I think the above is the better than... Well, what's the alternative? To have a Scrum Product Owner or a Scrum Master that for various reasons aren't focusing 100% on their roles.

Monday, October 13, 2008

"1 day remaining" forever

Let’s say the team is working on some story and need to finish a task which is originally estimated to 4 days. Once the team starts working they continuously decrease the estimate until they reach 1. The task then remains on “1 day remaining” for several days – even though the team actually do work (and hard, too) on finishing the task. You ask the team what’s going on, why the task never reach 0, and the reply you get is “Well, things keep popping up that we didn’t think of which has to be done too in order to finish this task. We can’t finish the task unless we also handle those things.”. Let’s say you don’t only see this occasionally. You see it over and over again, Story after Story and Sprint after Sprint.

What is the problem? (Please leave me some comments below if you’ve seen this problem). I have some likely suspects:

Estimation technique
I.e. your estimation process. If you keep having this problem then you probably need to consider if it is because you keep missing something in the estimation. Maybe it is always the same “thing” that keeps holding the task up from being finished? Could you incorporate that “thing” as an item in a checklist which is always considered during estimation meetings? Do you have the right people attending the estimation meetings? Do you have someone with deep enough, and right, knowledge about the system and technical domain to give his/her input to the estimate?

Technical Debt
As a system ages, the shape of its code, architecture, design etc deteriorates – its natural, and every coder knows it. The problem is that the rate of decay is so much higher if the people working on the system continuously take shortcuts. There’s a saying that I keep coming back to; “The dirty remains long after the quick is forgotten”. Think of that “dirt” as taking a loan. Eventually you will have to pay it back, and in the meantime you will have to pay interest on the remainder. “Interest” in our world is in the form of us having to deal with problems when developing new things; things inevitably take longer because the system is so complex. Paying back the loan will mean having to take time to eliminate those shortcuts (that dirt).

If your team tends to stay on “1 day remaining” for ages then you’ll need to consider if the problems they have (the unknown stuff that appears) are due to technical debt. If it is, you’ll want to prioritize very high some efforts for paying back some loans. There’s a business reality out there so, most often is simply not possible to halt development for six months and have all resources work only on refactoring. But at least you can ask yourself; are we increasing the debt still as we build new stuff? And if so, why? Most likely the developers do not want to take shortcuts, but they are more or less forced to because of pressure (implicit or explicit) from stakeholders. You’ll want to look at your team’s environment and see if you can somehow remove the feeling of them not being allowed to take longer time to do things “right”.

A mechanism to emphasize this is to introduce a bullet to your Definition-of-Done: “Not increased technical debt”. It should mean that it is not OK for a task/story to be finished until the implementation is such that it doesn’t increase the technical debt. I.e. it WILL inevitably take longer to finish, but the shape of the code will be better than otherwise. Next step is to replace that bullet with the following: “Decreased technical debt”. That will mean that with every story that is finished there has been some effort put in to refactor (even if ever so slightly) what’s already there.

Formulation of User Stories
If your stories are formulated in a bad way then the team will definitely have a hard time finishing. Maybe the stories are too big? Maybe there’s too much detail and no room for flexibility? If your stories contain every little detail then you’re really back to waterfall again. You’ll spend enormous amounts of time writing specifications, leaving no room for the developers to think for themselves and no way for them to build it in any other way than what’s specified. And hence, it will be naturally hard to develop in iterations. The problem is that the flexibility of the scope has been lost. You essentially end up with a fixed scope.

“Fixating a scope”, to me, is OK. Wow, that’s a controversial statement. Let me try and explain what I mean. As long as it is done on Theme/Epic level (i.e. something along the lines of what’s traditionally called “Function Specification” level) – and not on Story level (i.e. not like a Requirement Specification), then it is OK for it to be considered fixed. If you have someone (e.g. a customer) ordering something (e.g. a feature or a whole system) then it’s OK for that person to really think through what he/she expects. It’s even OK if that is written down.

But as the feature(s) come closer to the development team then it is important that it (however detailed) is broken down into User Stories which are flexible enough to be implemented in iterations, so that you are able to deliver something working at the end of every sprint.

If you’re the Scrum Product Owner, then spend some time thinking about how the stories are formulated. Is the level of detail “just right”? Is there enough flexibility in there for the team to be able to finish it (i.e. without being extremely restricted to an exact certain scope)?

Badly overlapping User Stories
Obviously there’s a million ways to derive (Mike Cohn calls it “trawl”, by the way) User Stories from a vision of a feature or system. I’ve stated previously that I believe the approach should be to divide the feature(s) into verticals (we call them “slices”). Regardless of if you listen to my advice or not :-), you risk running into the problem with overlapping stories. Note that I do not mean “overlapping” as in overlapping code/subsystem: on the contrary, continuous refactoring is all about iterating and reiterating the same code over and over again. What I mean is overlapping functionality; the problem comes when you can’t finish a Story because it intertwines with other Stories, in a way that you can’t tell which story you’re really working on. That is a badly overlapping User Story.

If the team keeps remaining on 1 day, then you should consider if maybe badly overlapping stories is contributing. Try reformulating stories with this in mind. Can you think of other ways to write the stories? Think of the acronym “INVEST” as a rule of thumb; stories should be
  • Independent
  • Negotiable
  • Valuable
  • Estimateable
  • Small
  • Testable
Finished does not mean Complete
This is, I think, as much a result of all of the above as it is a fundamental attitude in the people working on and around the system. When a Story is “Finished” it doesn’t mean that what has been done is in any way “Complete”. I keep coming back to it: Continuous refactoring, iterative development – this is what it is all about.

When unforeseen problems appear and a team stay on “1 day remaining” for ages, can it be that they put the wrong meaning in the word “Finished”?

“Finished” should mean that the story is implemented in some way. It doesn’t mean that the implementation has to be perfect, and it doesn’t mean it needs to be completed and never again returned to. I’ll post a separate blog some day about whether or not this means increasing technical debt or not (in my opinion it does not).

This, I think, requires a change in attitude among the developers, the product owner and the customer/stakeholders. A sprint demo/review doesn’t mean showing Complete stuff. It means showing Stories finished. “Finished” as in “Done-Done”, yes. But that doesn’t mean that we don’t go back immediately next sprint and add, change or remove stuff in order to finish another Story.

Also, another related cause may be that the sprint is too long. The longer the sprint, the more and the heavier the feedback will be from stakeholders. The more seldom the stakeholders see the result of a sprint the more they will feel that they have to “get all the comments out there” and thereby the bigger deal the sprint review will be to the team. And the bigger deal the more it implies “Complete”. So, consider shortening the sprint length (a lot) and get the customer and other stakeholders involved much more often. It is a matter of building confidence between team and customer – confidence that the result in the end is what was expected (or actually even better).

Thursday, October 9, 2008

Beyond the buzzwords

What is “Agile”? And what is “Scrum”? And “XP”? Oh.. And Lean? What is that? Can you “work with” Agile? Or what exactly is it that you do? Are all just different and c00l buzzwords meaning the same thing?

Oh, the confusion… =)

Well, I think of Agile as a “value package”. The word “Agile” is simply the name of a set of core values and principles about how “development projects” is viewed. So, you can’t really “work with Agile”, but you can have processes and project methodologies that are Agile – they fit the Agile core values. Here’s what’s written on Wikipedia about Agile Software Development: “Agile methodologies generally promote: A project management process that encourages frequent inspection and adaptation; a leadership philosophy that encourages team work, self-organization and accountability; a set of engineering best practices that allow for rapid delivery of high-quality software; and a business approach that aligns development with customer needs and company goals.”. There’s a manifesto that covers the core values, which you can find here: http://agilemanifesto.org/.

So, “Scrum” then, what is that? Is that a process? Or a is it also just a “value package”? No, to both questions. Scrum is an agile "process framework". It is not a process, it is a framework for processes to exist (be formed) within. Scrum is an interpretation of Agile Development. Right or wrong, I view Scrum as an “instance” of an Agile development approach. Scrum sets the boundaries and the general direction for the processes but it doesn’t state exactly what those processes should be or what steps and methods they need to hold. That is why it is said that if you “implement scrum” you have to find your own way: you have to create your own set of processes and practices that fits your organization and the Scrum framework. It is quite correct to say that you “do Scrum”: it means that you have identified and enforced a set of practices that cover all the aspects of Scrum. You can read more about Scrum on (lots of places) for example http://www.scrumalliance.com/ and on Wikipedia.

Ok, and “XP” (eXtreme Programming)? Just like Scrum, XP is an “instance” of an Agile development approach. So XP too needs interpretation and adaptation to be implemented in your organization; it is not an out-of-the-box ready set of processes. The difference between Scrum and XP is that Scrum is more focused on project management and team aspects of software development, while XP focuses almost entirely on the hands-on development practices. Scrum for example talks about how to plan your iterations, using Sprint Planning meetings and Sprint Reviews. It doesn’t say much about how to actually write code. XP on the other hand doesn’t say as much about managing your project (although there is some overlap) but focuses much more on how the developers need to sit together and work in pairs, or that you need to have continuous integration, automatic testing, Test Driven Development (TDD), and so on. Scrum and XP fits really nicely together, and together they form a great and whole-covering framework for a really agile development approach. In our organization we have yet to adopt XP to any extent. But we’re getting there.
You can read more about XP on (lots of places) for example Wikipedia.

And finally, what is “Lean” in all of this? Well, it too is a way of observing the world - the world of manufacturing to be precise. According to Wikipedia, Lean is “the practice of a theory of production that considers the expenditure of resources for any means other than the creation of value for the presumed customer to be wasteful, and thus a target for elimination.”. It originates from the automotive industry (e.g. Toyota in the 1950s) and was later adapted to software development (Lean Software Development) by Mary and Tom Poppendieck. Just like Agile it covers a set of core values, and sets a direction for how to go about “manufacturing”, in our case Software Development. Lean is all about spotting and removing sources of “waste”, decreasing cycle time, continuous improvement, respect for and personal growth of individuals, etc. Lean and Agile fits very nicely together.

Confused? Yeah, me too ;-)

Wednesday, October 8, 2008

Slices (Verticals), User Stories and Scrum

We try to divvy our stories in a way that they span from top to bottom; through all layers of the architecture. We call such an isolated part of a function or project, a “Slice”. It is also commonly known as “Vertical”.

But why develop in slices, or verticals, why even care about it, why not just do the User Story regardless and forget about if it is a horizontal or vertical or whatever…? Well, we’ve tried various approaches here. The one thing I am certain of is that this is hard. It requires skill to be able to produce “good” user stories. And not only is it hard, it is also very important: because badly formed User Stories can wreck velocity. Properly shaped and formulated User Stories is one of the basics, a requirement, for a well-functioning sprint execution. So, Hard + Important means that we have to focus on learning it and really spend time thinking about it – and continuously evaluate our approach and view of the matter.

Ok, so then why vertical and not horizontal?
If you do things horizontally, isolated to each architectural layer, then it will be hard for you to actually deliver working, testable code. By doing Slices you are forced to deal with every layer right from day 1. This foundation will then be built further upon, as you iterate the software with every new slice. Also, working vertically – meaning involving every layer in every slice – will often require different competencies (for example UI/web, C++ and SQL/db). Suits nicely with our cross-functional Scrum teams, doesn’t it?

To use Lean terms, it’s about decreasing the “batch size” and ultimately minimizing cycle time; i.e. the time it takes until you deliver something valuable to the customer. If you work horizontally it will take longer until the customer has something that brings value. A complete user interface, but empty of all functionality, is rarely valuable to the customer (and rarely “complete” for that matter, until you’ve built the underlying stuff). And it is definitely not something “releasable”. Remember that you want to strive towards having something releasable every sprint. Sometimes (often?) it doesn’t make sense to release “half a feature” (e.g. “Add User” without “Edit User” functions), but at least you need provide that option and let the customer decide what he wants and doesn’t want: don’t make the decision for him by implementing horizontally.

So what forces are in play here, with regard to stories, architecture and Scrum? Well, let’s recap;
  • We’ve just concluded that we have a requirement to build vertical increments of the system: Slices.
  • We also have a requirement to “finish” (Done-Done) things, with each sprint.
  • We even have a requirement to be able to “Release” at the end of every sprint.
  • And we have a requirement to think iteratively, i.e. we should come back continuously and refactor our code (continuous refactoring – see blog post below)

Tuesday, October 7, 2008

Should the Actual Velocity measurement include “Unplanned stuff”?

A question that keeps popping up here is whether or not the Actual Velocity (i.e. the sum of the Story Point values of all Done-Done stories from a sprint) should include “Unplanned stuff”. I.e. things that were not expected and which are not really part of the Sprint Backlog. The question assumes that the team somehow estimate (in Story Points) that “unplanned stuff” as it appears, and put it into the Sprint Backlog, and consequently track it to Done-Done state and as a consequence include it in the Actual Velocity measurement.

First of all, I think the way to go is to minimize the amount of unplanned stuff. Period. If you have lots of unplanned stuff then you should consider shortening the sprint length, or possibly look at how well the Product Backlog is managed and prioritized.

But there may still be unplanned stuff that appears. There’s no way around it. It’s a reality and we have to be agile enough to be able to handle it.

Let me just emphasize that if you were to estimate the unplanned stuff, you would have to estimate it using the same unit and estimation reference as the regular User Stories; otherwise the Story Point values would not be comparable or possible to sum up. In our case we estimate all our User Stories using “Gnome days” (se blog posts below), so the Story Point value of the unplanned stuff would have to be estimated using that unit too.

So in every sprint we have two alternatives, right: either you do include unplanned stuff (in Actual Velocity) or you don’t. Let’s take an example where we have a team doing a sprint starting out with 30 SP of Estimated Velocity. Let’s also assume that the team so far has had no unplanned stuff appearing in previous sprints and the Estimated Velocity of 30 is pretty accurate and stable.

Case 1: the team decides to include “unplanned stuff” in the Actual Velocity measurement
The team works for a sprint and there’s a bunch of “unplanned stuff” that appears. The team estimates it and includes it in the Actual Velocity measurement at the end of the sprint. The team’s Actual Velocity will probably be close to the Estimated Velocity, even though parts of the work Done-Done is actually “unplanned stuff” that has replaced User Stories that were originally part of the sprint backlog. As a consequence the team will have completed less User Stories than expected, because it had to deal with that unplanned stuff. So the understanding of the team’s velocity at the beginning of the sprint has proven incorrect, in the sense that it incorrectly indicates that the team can perform 30 SP of User Stories from the Product Backlog. That is true only if there’s no new “unplanned stuff”.

Case 2: The team does not include “unplanned stuff”
The team works for a sprint and the unplanned stuff that appears during the sprint is not included in the velocity measurement, so when the sprint ends the team’s Actual Velocity (25 SP for example) will be lower than the Estimated Velocity (30 SP in the example above) – because the team was distracted from the original sprint backlog doing that unplanned stuff. As a consequence, on the next sprint planning the team will observe its past Actual Velocity (25 Story Points in this example) and they will thereby commit to less. This way, they automatically adjust for a “similar amount” of unplanned stuff to appear in future sprints.

Conclusions
Given that you have a relatively steady inflow of unplanned stuff (as opposed to very rare occurrences, or spikes) in your sprints then you do not want to include Unplanned Stuff in your Actual Velocity, because it decreases the quality/reliability of the measurements. As a consequence you risk constantly taking in more User Stories than you’re actually able to complete.

Note that I’m not saying that you shouldn’t estimate or do the “unplanned stuff”. If there’s unplanned stuff that appears that has to be done, then sure you can include it in your scrum board, estimate the remaining time and track it in the burndown. But do not give it a Story Point value and do not include it in the measurement of Actual Velocity.

Monday, October 6, 2008

What metrics should be collected and how to measure and use "Velocity"?

What should be measured during a sprint? Obviously, you can measure a million different things. My suggestion below is tilted towards the simplistic approach. I like to keep measurements and maths to a minimum. Don’t get me wrong; I love Excel, metrics and complicated statistical formulas... But as far as the team goes, and what they need to care about and focus on, I suggest sparing them the nittybitty details and just collect the bare essentials.

  • Estimated Velocity: The sum of the story point value of all those User Stories taken into the sprint backlog (at start of sprint). You can read some more about the concept of "Velocity" here, on the Scrum Alliance site.
  • Actual Velocity: The sum of the story point value of all those User Stories that are determined to be Done-Done at end of sprint.
  • Average Actual Velocity: This should be the Actual Velocity averaged out over a number of sprints (provided the team i relatively stable – or this will have little or no meaning).
  • Mandays in the sprint: This is simply the number of persons in the team multiplied by the number of workdays available in the sprint.
  • Focus Factor: The Actual Velocity divided by the number of Mandays in the sprint.

Example: Let’s say I have a team consisting of 4 full-time individuals and one person working half-time with some other duties. Let’s say they work in 2 week sprints. Let’s further say that the team takes in a total of 35 story points at sprint start. Let’s say they manage to complete (Done-Done) a total of 28 story points. So;
Estimated Velocity = 35
Actual Velocity = 28
Average Actual Velocity = Avg(28) (So far only 1 sprint to average over)
Mandays in sprint = 4.5 persons * 10 days = 45.
Focus Factor = 28 / 45 = ~0.6.

So, what values do you use during Sprint Planning? I suggest the following:
  • Average Actual Velocity, or at least last sprint’s Actual Velocity, to compare the new sprint’s Estimated Velocity to, as a sanity check.
  • Probable Velocity = Mandays available in the new sprint X Focus Factor.
Example (continued from the example above): If the team and the sprint lengths remain unchanged then the team should probably not take in more than approximately 28-30 story points into the new sprint. If the new sprint is, for example, 15 days instead, you can use the previously measured Focus Factor to calculate the probable velocity; (4.5 persons * 15 days = 67.5 mandays) * 0.6 FocusFactor = ~41 story points.

The most important thing is that the team feels comfortable with the commitment on the sprint planning. The sum of the stories selected by the team should however be verified against previous velocity to make sure that the team hasn’t substantially under- or over-committed.

A team’s velocity is a measurement of the reality. It can’t be “too low” because the reality is what it is; making up excuses to why an actual velocity is lower than its estimated is not meaningful. If you see an actual velocity that is lower than your estimated then you will want to lower your estimated velocity next sprint. “But”, you say, “the velocity needs to be higher, it is too low, the project will take too long”. … Yeah, right. But it is not the velocity that needs to be higher, it is the environment in which the team operates which needs to be different. You will only fool yourself (and the stakeholders) if you make up excuses and count on an optimistic, higher velocity than your actual; all you’ll manage to do is to make your plan look good, but it will then – inevitably – be a lousy, incorrect, optimistic and unrealistic plan. The one thing (except resources) that affects velocity is impediments. Remove impediments and the team’s velocity will go up. This should be your approach if you feel the velocity “is too low”.

Friday, October 3, 2008

Scrum Master role vs Leadership vs Project manager

Is the Scrum Master "a leader"? Yes, an informal one - not a formal one. The Scrum Master should not perceive the role as a leader role. The Scrum Master should be someone the team trusts and looks up to - but should be someone within the team, on the same hierarchical level as the rest of the team members.

I'm pointing this out because I have noticed that being appointed "Scrum Master" may imply "becoming the team leader". Which is incorrect. I see the Scrum Master as a representative for the team - the "curling parent". A person sweeping a clean path for the team to go forward. That does not mean leading per se.

If the Scrum Master is "the leader" I think you risk losing some of the cooperation effect and the "tightness" of the team, and with a strong leading Scrum Master the team risks relying too much on that single person to take decisions and lead the way. Or at least it will be easy for the Scrum Master to fall back in old tracks and take charge: thereby undermining the self-organization and collective committment of the team.

So, when you appoint Scrum Masters, make sure it's someone within the team, who the team trusts, knows and looks up to. Most often you'll find there is one or a few informal leaders in the team, and either of those would be perfect for the Scrum Master role.

An important note here is that the Scrum Master should not be the tech lead of the team. Don't make the Expert the Scrum Master. Why? Because the Scrum Master's first and foremost priority will be to remove impediments and be "Scrum Police" (sorry, I mean "Advocate" :)). That means taking time away from other tasks, such as coding. So if you put your tech lead as Scrum Master you will lose pace and you will hinder the rest of the team (because the tech lead will be busy removing impediments and won't have as much time to help others).

Finally, a pitfall which we fell into - which I guess is not totally uncommon for organizations and people who are new to Scrum - is to mistake the Scrum Master role with the classic project manager role. When we started our Scrum implementation we thought that the project manager (called "Producer" in our world) should be the Scrum Master. That was indeed a mistake. Just read the above :-). A lesson learned here is that the project manager role is sortof altered and spread out across Scrum Master and Scrum Product Owner. In our implementation, we regard the Producer as the "Scrum Product Owner" nowdays. I'll write some more about that in future posts.

Thursday, October 2, 2008

Don't document more than that you have to ask to understand

Jeff Sutherland writes in his most recent blog entry about Agile Specifications. Interesting stuff!

I have rule of thumb that I use and try to teach everyone, when it comes to documentation: Don't document more than that you have to ask to understand. It is, I think, something to remember when it comes to writing agile specs, user stories, etc. Spending lots of time writing fancy specs and try to catch every little detail is just a waste of time, because I think it causes a false sense of trust in the spec. You won't succeed just because you discribe every little aspect of some feature. What will let you succeed is if the developers manage to understand it well enough to build it. And communication is key here. So don't write fancy specs just for the sake of it. Communicate instead! Ask! But as Jeff describes in his article above, of course you should document enough. We do write specs too (I'll describe our "scoping process" some day). But the above rule of thumb should always be kept in mind...

Picking stories at sprint planning = Drawing The Line

My thoughts today is about how the team picks stories at the sprint planning meeting.

One point of the sprint planning is for the team to be allowed to pick the amount of work they feel they can cope with, and commit to, for the coming sprint. As a preparation for the sprint planning meeting, the product owner goes over the backlog and revises the story prioritizations based on most recent feedback/input from sprint reviews etc (whatever affects the priorities).

Our approach up until recently was to let the product owner present and make available (e.g. on a whiteboard) the X topmost stories in the backlog. Those stories would be considered "up for grabs"on the sprint planning, but without relative priority. The team could freely pick any of them, in any order. It sounded so good on paper; the team has complete freedom over which stories they pick, yet the product owner has a say about it by means of him prioritizing the backlog prior to the sprint planning meeting.

The pitfall with this is that since the team picks any stories from the top of the product backlog, the stories enter the sprint backlog without priority; or at least the priority has less of a meaning now. And it is therefore tempting to just regard the set of stories in the sprint backlog as one big unordered set of stories - all which must be completed. And as a consequence you lose one of the most valuable concepts of Scrum & Agile: focus on the most important thing first. Plus, it leads to people in the team isolating themselves with one story each, which means the cooperation effect is lost.

Don't do what we did!

So how should you do it? Well, the product owner should prioritize the stories in the backlog, and they should be ready-ready (see previous blog post). On the sprint planning meeting these stories are put in a list and the team can not just pick any of them. Instead, what the team does is to draw a line somewhere. That's it. This drawing of The Line represents their committment. Everything on one side of the line is what they commit to delivering, and the rest is left for consequtive sprints; or whatever the product owner choose to do with them. In our case, we should have the stories printed out on index card papers, and put on a whiteboard from left to right in descending priority order. The team then draws a line using a whiteboard marker. Simple enough! :-)

Wednesday, October 1, 2008

Ready-ready: the Definition of Ready for User Stories going into sprint planning

You all know what “Definition of Done” means. Just for the hell of it I’ll recap it anyway; What it refers to is a state of a chunk of work. The state is also referred to as “Done-Done”. The purpose of the Done-Done mechanism in Scrum is to ensure that those things completed in the sprints really are completed in a commonly acceptable, controlled, premeditated and agreed sense. For example, nothing should be considered “completed” until it is e.g. tested, reviewed, committed and merged – e.g. “Releasable”. Without the definition of done we risk building up a mountain of stuff that we push ahead of us, which eventually (e.g. release-day) will kill us – or at least give us a hell of a fight. What’s even worse, without the definition of done we not only build up such a mountain of stuff, we also have no way of knowing how big that mountain is until we start getting our hands dirty and digging in. And what’s even worse is that many of the things inside that mountain will take so much longer to do later than if performed immediately day 0, because the longer it waits the harder it becomes to fix, to understand, to find, to remember, etc.

So, I think we’re all in agreement that we need a definition of done for the work we ship out of a sprint. Now I think we need a similar mechanism for the work taken in to the sprint. A mechanism which ensures that the stories taken in is really ready to be taken in. Let’s call this state “Ready-Ready”.

First of all, what does “taking in” mean? Well, it means “available and up for discussion on the sprint planning meeting”. So the definition of ready needs to describe a state of stories which they must be in, in order for them to be eligible for discussion on the sprint planning. Before they have reached the Ready-Ready state the stories do not exist and will not be considered for sprint planning. I.e. the top part of the product backlog has to contain only stories which are ready-ready.

Why? To save time of course: to spare the team tedious discussions about stuff that isn’t sufficiently thought-through. To let the team focus their valuable time and energy on things that are as “safe” as possible.

So, the definition of Ready should be;
  • A user story exists which points out the actor, describes the targeted feature and the purpose of it. The story should be formulated like this; As a I want to because . The first two are often easy to pinpoint but the purpose is often overlooked, because it feels obvious. But is it? The idea with writing down the purpose is to give the reader a good feel for the context of the feature – what problem it solves. Not only what it is but also why. This way, the reader can form his own opinion on how to implement it. It gives the reader a deeper understanding for the story.
  • The formulation of the user story is general enough for the team to have the flexibility of delivering it in increments. Don’t describe every detail. Describe roughly what you want but leave the rest for discussion.
  • The story is small enough to fit inside a sprint. Stories larger than that need to be broken down. Breaking down and reformulating stories have to be done before the sprint planning. Obviously, a story cannot be completed in a sprint if it is larger than a sprint. So care should be taken to ensure they are small enough.
  • Each story has its own unique priority in relation to every other story in the product backlog. This is to ensure that the team can work on the most important things first, and to force the product owner to make intelligent decisions about the order of stories.
  • The story has a description of how it can be tested (demoed). The description need to give the reader a good sense of what determines if the story is completed or not. It should be e.g. the acceptance test that is described here. It can be used as a description of how to demo the story too.
  • The story has been estimated (in story points). This should preferably be done by the whole or parts of the team, but at the very least it should be done by individuals with the same understanding of the system and the domain as the team who will eventually be working on it.
So in our case we should have meetings prior to the sprint planning meeting, where the product owner meets the teams and sits down and discuss the product backlog. Don’t forget to time-box the meeting. The goal of the meeting is to deem stories as ready-ready. It will require that some stories are broken down, reformulated, etc – and estimations are done using poker planning. A tester or test leader is required to attend here too, so that the formulation of the “how to demo” is really testable and meaningful.