Friday, November 27, 2009

What unit to use in burndown?

What unit do you use for plotting your Burndown Chart? Do you use “Remaining Hours” of the tasks? Or perhaps “Story Points” of the stories in the sprint backlog? Or something else?

Up until recently I’ve taught Scrum teams to just sum up the number of remaining hours for all tasks on their Scrum Board, and plot that sum on the burndown chart. That has been my default approach and I haven’t really experimented with it.

Why even bother breaking down a Story into Tasks?
The immediate answer to this question is of course that we do it to be able to sum up estimated Remaining Time. Duh :-). But I think there’s more to it, because a Story is more than just the sum of its Tasks. The real reason, I believe, is to get the team thinking; to get the team to reflect over what challenges the Story hides so that they maintain an as clear picture as possible of where the implementing is heading and how the sprint is progressing. This implies that breakdown of a story shouldn’t be done just once (e.g. at sprint planning). The view of what tasks a story contains changes as the implementation progresses and things are discovered. It’s natural that the contents of the story vary over time. Nothing of what I have written here is controversial. It’s all part of Scrum.

What happens in practice?
Plotting a burndown based on the sum of Remaining Time obviously requires that the team first breaks down the Stories into Tasks. This, in my experience, has always been somewhat of a challenge; for the team to actually end up with useful tasks. A good breakdown of a scope into User Stories is hard enough, and then to also successfully break a Story down into good Tasks … Well, it is hard and it requires lots of practice. I’ve always struggled getting teams to continuously reflect over their breakdown and about how to improve it. But in the end, tasks often seem to end up saying more or less meaningless things like “Implementation”, “Testing” and “Review”. Fail.

Failure in Task breakdown like this effectively prohibits the team from having any use of it. A breakdown like this doesn’t say much about the Story and consequently there is little or no use in continuously reviewing the breakdown to try to learn more about the Story as implementation progresses. The only remaining purpose of the breakdown, then, is to force a daily re-estimation, so that numbers can be summed up and plotted on the chart. The burndown and the estimation of Remaining Time becomes an end in itself.

We started counting "Number of remaining tasks"
My team, being in the situation described above, felt that the daily task re-estimation was (more or less) meaningless, and proposed that they should try and completely skip the estimation all together. If nothing else, it would save them that estimation-time that was borderline waste anyway. But they also realized that the burndown has an important purpose (yesterday’s weather, indicating progress, early indication of problems, etc) so they felt that they still needed burndown. But if they don’t estimate Remaining Time, then what do they plot on the burndown chart? Well, they agreed to try to plot the number of remaining (not done) Tasks. Simple! And not completely new; it’s a fairly common practice, if I’m not mistaken.

Interesting consequences of this change
This change in the process had a very interesting side-effect: all of a sudden the Task breakdown produced a large number of highly meaningful tasks. Instead of just saying “Implementation” the tasks that the team now produce actually describe what needs to be done, in pretty deep detail. The number of tasks produced both at the sprint planning and on the Standups is significantly higher now than before, and they say a lot more about what needs to be done.
It just seem to come more naturally now. The only difference is that we no longer estimate the tasks afterwards. That’s it. I think this is very interesting. It’s like if there was a mental block before, that has now been removed once the team doesnt have to bother estimating the tasks. I don’t know, but whatever the reason, I think it's great. The team is satisfied with the change. I feel that the quality of the task breakdown of the Sprint backlog is much higher now, and consequently the understanding of the team’s progress is much better (both for themselves and for me as a stakeholder). The project is more transparent.

The drawback of this change might be that the burndown probably is a little less accurate, because the size of tasks vary a lot. On the other hand, how accurate was the burndowns before, when we were estimating remaining time almost as an end in itself? Hopefully the accuracy now is not much worse. And if it is, I think it is outweighed by the gain in insight and increased feeling of progress.

Let me know how you work with tasks, and why… As always I’m curious to get some feedback.

Friday, November 20, 2009

Agile projects are never late

In the post I want to talk alittle about "late projects". It is a topic that is somewhat current for me and I have spent some time pondering about this phenomenon :-).

The question I ask is: What does it mean for a project to “be late”?

Well, in my view, in order to “be late” there has to be a target-date or a target-state (e.g. deadline or a certain scope or both) that the project’s current state is compared against, and the result of the comparison is that the state is such that either expected date or expected scope isn’t achievable.

If we can agree that this would be some reasonably valid definition of “being late” then it implies that the following factors are involved in “becoming late”:
• Date
• Scope

When a project “is late” it means that (for whatever reason) what was expected to be ready at a given time, isn’t. Date doesn't match scope - or vice versa.

Now, let’s assume that I fix the date, i.e. I commit to delivering my software on a certain date, but I don’t commit to an exact scope by that date. Instead, I make sure that the scope is allowed to flex so that I can deliver only whatever is ready by the time I reach the date that I have committed to. As you all know by now this is what Agile is all about. It is OK to commit to either date or scope, but never both.

Is it now possible for my Agile project to “be late”?

And I believe the answer is "No". An Agile project can never “be late”, because it is impossible. “Late” is a concept that does not apply to Agile projects. By allowing the scope to flex you effectively remove one of the two required factors for becoming late in the first place. Consequently; if the project is late, then it is not an Agile project.

One could of course argue that “being late” can also be when a stakeholder desires the scope to be in a certain state sooner than it actually is. However, in my book this is called “wishful thinking” on the behalf of the stakeholder, and it has nothing to do with the project “being late”. That needs to be handled by proper management of expectations – which Agile and Scrum projects in particular solve by opening up the project with full transparency to stakeholders (velocity measurements, release plan based on velocity, highly visible burndown charts, deep involvement of customers and other stakeholders, short feedback loops with regular deliveries and demos, etc).

What the stakeholder above has to realize is that the scope will be delivered as fast as reality allows. The state the project will be in on any given date is determined largely by the “context” in which the project is being realized. When I say “context” I mean all those factors that are affecting how fast the realization can happen: the competence of team, the cooperation of team members, the availability and effectiveness of tools used, the ability for the team to focus on the task (i.e. disturbances), the mood & motivation of individual team members, the availability of information, the clarity/understanding of what is requested, and so on.

So, let’s agree to never again claim that an Agile project is “late”.

OK? :-)

Thursday, November 5, 2009

Scrum. You're doing it wrong when...

…your sprint is so long that you feel you have to make a time-plan for it.
…your Scrum Master tells people in the team what to do and when.
…the team report their progress to the Scrum Master.
…estimates are not done collectively by the team-members who will be doing the work.
...you hear one team-member say to another; “Sorry, I’m really focusing on my own tasks here in order to finish by the end of the sprint, so you will have to solve your own problem yourself.”.
…all stories in the sprint finish on the last day of the sprint.
…lower-priority stories in the sprint gets done before higher-priority ones.
…the team has as many stories in progress as it has team-members.
…the Scrum Product Owner can’t explain every single story in both the Product Backlog and the Sprint Backlog.
…you still think it’s a good idea to write down all requirements before you start coding.
…you consider customer feedback a risk.
…the team doesn’t know who their Scrum Master is.
…your system testing happens all at once, after a couple of sprints.
…you still think that a detailed Gantt chart is a useful planning tool for the software development tasks.
…your sprints are 6 weeks long.
…the team’s burn-down has been flat-lined for three days and they’re not discussing why.
…you think the customer knows what he wants.
…you think the team knows how to build what the customer says he wants.
…the team doesn’t know how much work is remaining of the sprint.
…you think it’s OK if the team has 45-minute Daily Scrum meetings.
…you come running to the team after a sprint and yell “Hey, we have to release now!” and someone in the team responds “…Oh, but wait, we still haven’t done any system testing… And there’s some more code review to be done too.”.
…the team doesn’t know when and where the sprint demo will be held.
…the team constantly over-commits.
…you think it’s OK that the team has their Daily Scrum meetings every other day instead; “It’s their choice, and meetings take so long, so….”.
…the team is persuaded to add more to the sprint than they really want.
…the team accepts to include stories that have not been estimated.
…you don’t collect feedback from the customer after every sprint.
…the team doesn’t know who their Scrum Product Owner is.
…your team is larger than 8 team members.
…the team doesn’t know what the sprint goal(s) are.
…your Scrum Product Owner misses a sprint demo or a sprint planning every now and then.
…the team decides (or doesn’t follow or care about) the order of the stories in the sprint backlog.
…you think the Scrum Master is sort of like a Project Manager.
… you think Scrum doesn’t require or need planning.
… you think you can’t have a deadline or commit to a scope "because you’re using Scrum".
… you think you don’t need to document "because you're using Scrum".
… you commit both to a certain scope and to a certain date.
...you need to add stuff to the sprint half-way through.
…you think the Scrum Product Owner is sort of like a Project Manager.
…the team doesn’t know their velocity, or the meaning of it.
…the team haven’t met the Scrum Product Owner for a week.
…the contents of a sprint demo comes as a surprise to the Scrum Product Owner.
…the team doesn’t know or understand the overall vision of what they’re working towards.
…you can’t release the software after every sprint.
...Scrum doesn't work.

Tuesday, October 27, 2009

Follow-up on the Scrum Practitioners meeting Oct 15th

Thank you everyone who participated on the Scrum Practitioners meeting on the 15th of October at Axis Communications in Lund.

Some of the things discussed were;
  • Distributed Scrum; does it work? A few of the participants had some experience in this, and the opinions were mixed. Bottom line is; yes it is certainly possible to get it to work. I.e. it doesnt automatically fail. It requires work, though. But that is true for any methodology; if you have team members on a distance you have to deal with the overhead cost in terms of extra communication, tools, activities, travel, whatever. Depending on how the distribution looks you can try different approaches; divide into teams per site, or distribute teams and work with conference solutions, digital whiteboards, etc for the daily meetings. Experiment. Distributed teams and distributed Scrum is not ideal, but if you're faced with a situation where you have a distributed team, don't automatically rule out using Scrum.
  • How/if sort out design-issues before sprint start? The problem discussed was that of things taking a long time to finish if you do "design stories" in one sprint to discover what/how to implement, and actually implement in in the following sprint. With 4 week sprints it can take you 8 weeks to finish a story, even if it is not that big. The concept of "spikes" was discussed, i.e. prototyping and delivering a piece of working and valuable code in each sprint, rathen than doing "only design" separated from "only development". If you do "design stories" it may be wise to timebox rather than trying to estimate it, because the definition of done for "design" is hard to grasp. Another tip can be to consider shortening the sprints, to e.g. 2-3 weeks instead of 4.
  • The problem of traditional, conservative customers, demaning fixed-price-fixed-scope contracts was discussed. The concept of "Money for Nothing and Change for Free" was discussed as an interesting theory. None of the participants had experience with that approach, though.
  • Coping with less-than-fulltime resources in a Scrum Team: The problem of traditional customers and product owners who do not realize that Scrum requires a much higher degree of involvement. Not sure we came to any intelligent conclusions on how to address this problem, other than the importance of realizing that it is a problem if the customer/PO is not involved.
  • Problems with large team size was discussed, and consequently the problem of daily scrums taking too long time.
  • Is the traditional project manager automatically the best choice for Scrum Master when introducing Scrum? Not necessarily, was the conclusion. This seems to be the default choice in the industry, but it is not necessarily the best. It depends on your organization, the people, the problem domain, etc. The Scrum Master may be a good choice for the Project Manager, but Scrum Product Owner may be too.
Several other interesting things were discussed too, but I will not go into more details here.

I'm looking forward to the next meeting.

Monday, October 5, 2009

October meeting Scrum Practitioners South of Sweden

I'm planning the October meeting for the network Scrum Practitioners South of Sweden. I'll update this blog post as details are decided. For now, the meeting is planned for Thursday October 15. It will be between 17:00 and 19:00, hosted by myself, in Axis Communications' office at Emdalavägen 14 in Lund.

The topics of discussion are (under construction ;-)):

* Distributed Scrum; Does it work?
* How/if sort out design-issues before sprint start?
* How do I get the product owner more involved?
* Coping with less-than-fulltime resources in a Scrum Team
* Is the Project Manager the best choice for Scrum Master when moving from a Traditional to a Scrum organization? Who decides, and who is the best choice?
* Is the "Product Manager" the best choice for "Scrum Product Owner"?
* Story formulation, estimation, story breakdown into tasks and prioritization
* ...<insert topic here>

Wednesday, September 16, 2009

Follow-up on the Scrum Practitioners meeting Sept 7th

To follow up on our first meeting in the Scrum Practitioners South of Sweden network;

I don't intend to recap all the discussions and conclusions on the meeting, but rather I just wanted to briefly list the major topics discussed as I recall.

The meeting was held as planned at Labs2 in Lund with a total of 8 participants including Mathias Thinsz (Product Manager & Scrum Product Owner at Labs2) who was our host.

Mathias went through and explained the major parts of Labs2's Scrum implementation. It seems to me that they have a very solid and healthy view on how they development software. It's a clean and well-thought-through implementation of Scrum. A great example on how well Scrum works.

Related topics discussed were;
* How Labs2 use JIRA and a custom developed web application for managing the backlog
* How and when Labs2 estimate their work in the backlog, and how they use the storypoints unit
* How Labs2 has divided their development organization
* How and why Labs2 have divided their development time into fixed-length iterations
* How Mathias (as Scrum Product Owner) prepares for a new sprint and who is involved when
* How and why Labs2 use "Sprint Goals" as a major tool for directing development inside a sprint

There were tons of other interesting questions and discussions as well, but too much for me to go into here.

The general feeling, in my understanding, was that the meeting was interesting and inspiring and everyone had a lot of questions and topics which they wanted to continue discussing.
The number of participants was optimal and the length of the meeting as well. Hence; my conclusion is that there is a definate interest for meeting again and that the next meeting should be in the same format.

I want to take the opportunity to thank everyone who joined and not least Mathias Thinsz and Labs2 for the openess with how they work!

Friday, September 4, 2009

Mon Sep 09: Meeting Scrum Practitioners South of Sweden

On Monday the 7th of September 17:00-19:00 we're a bunch of people meeting at Labs2's office in Lund (Sweden :)) to discuss how Labs2 are working with Scrum. Anyone intersted in joining and sharing experiences is welcome.

Join the LinkedIn group "Scrum Practitioners South of Sweden" and post your interest if you want to join the meeting.

My idea with this is to get software professionals in the Öresund region to meet informally, network, and not least share experiences and help eachother improve our way of working with agile software development. Meetings will be hosted by a different group member each time, and the discussion theme will vary from meeting to meeting.

See you there!

Thursday, September 3, 2009

Scrum but... test

This has been around for some time, but if you haven't taken it already then do it. It's a good way to compare yourself to others and help emphasize your points of improvement.

Go to the Scrumbut test!

I just took the test and my team currently scores 7.1 out of 9.

Monday, July 20, 2009

Estimating & prioritizing the product backlog

One of the things that I’ve been struggling with getting my head around since I started using Scrum is; how much effort do I put into estimations of the product backlog, and do I or don’t I estimate all at once? And do I re-estimate stories as I go along and learn more?

Originally I’ve been under the impression that I shouldn’t attempt to estimate it all up front. It's easy to argue that it is rather un-agile to do so: because I would be spending effort on low priority stuff, which is a big risk for waste. On the other hand, if I don’t estimate the backlog up front, I can’t tell its entire size and consequently I can’t (potentially) tell what I will have ready by when.

So far, I’ve been trying a couple of different approaches. One of my favorites has been to estimate in priority order up to a few months worth of stories (a "handful" of sprints). The remaining unestimated stories I apply an average size to based on the estimated ones. I know, it’s risky, but it’s at least something. And obviously, as time passes, I estimate more and more; trying to keep about 2-3 months worth of stories estimated at all times. For some time this has been my approach to release planning without estimating everything up front.

A problem I only recently realized I’ve had is that this approach requires me to prioritize before estimating; and consequently my prioritizations can’t take size into consideration. My prioritization has only been made based on “value” (whatever that is) without regard to cost. Is that optimal? I think the answer is No. Given I have a choice prioritization should take size into account. Or at least, if I don’t take it into account it needs to be a conscious decision not to. Which brings me back to my original question; how do I do it, do I estimate everything up front or not?

Recently, I’ve decided to try changing my approach (inspect and adapt, right?) and actually estimate the entire backlog up front, before prioritization. The good thing is that estimating in story points is very quick (the team usually doesn’t have to spend more than a few minutes per item) so the time spent on (potentially) low-priority items is minimal.

Once I have the story point estimates of all backlog items I use a prioritization technique called “Theme Scoring”. Let me try to describe it with a simple example;

The first step is to decide on a couple of aspects ("selection criteria") to judge stories on; for example “Improves user interface”, “Simplifies maintenance”, “Brings revenue in Q3” and so on. The combined criteria will represent what you call “business value” so whatever you put in that term should in one way or another be reflected by those criteria that you pick. Try, however, to keep it rather simple e.g. up to a maximum of 4-6 criteria or otherwise your scoring will become very tedious and the effort you spend on scoring (and consequently the risk of waste) I think will be too great compared to the benefit.

Once you’ve decided on your criteria you need to weigh them relative to each other. Maybe “Brings revenue in Q3” is the most important criteria to you right now, so let’s give it the weight of (for example) 5. “Simplifies maintenance” is not that important right now so you give it the weight 2 (it’s less than half as important as the revenue criteria). “Improves user interface” is almost as important as the revenue aspect, so you assign it the weight 4. Here are your criteria and their weights so far; “Brings revenue in Q3” = 5, “Improves user interface” = 4 and “Simplifies maintenance” = 2.

The next step is to go through all of your backlog items, one by one, and score them on a scale from 1-5 in terms of how they affect each of your selected criteria. For each criteria you can usually identify a “baseline story”; a story that gives you a fair amount of value in terms of that criteria, yielding in a 3 point score on that 1-5 scale. There should be stories that yield in a higher score as well as lower. Stories should be compared to that baseline story for each criteria to determine if it should get a higher or a lower score than the baseline.

This could be your result after going over all criterias for all stories;


The numbers marked red are the baseline stories for each criteria. The black numbers in the criteria columns are grades I assign each story on a scale from 1-5 relative to the baseline. 1 and 2 means much lower and lower than the baseline, and 4 and 5 means higher and much higher than the baseline. A grade of 0 means that the criteria doesn’t apply to the story. The Score is simply your grade multiplied by the weight, for each criteria.

Given your selected criteria and their weights, in the example here Story 3 is the one giving you the most value (37), followed by Story 2 (18) and then Story 1 (17), Story 5 (8) and Story 4 (6).

Now, it is time to compare value to cost (estimated size in Story Points);


So far this technique has been possible for me to do all along, even while I have been lacking estimates of parts of the backlog. But the following part – taking cost into the equation – obviously requires me to have the estimates. Putting the Benefit in perspective to Cost is a great idea; it gives me the opportunity to maximize benefit while minimizing cost.

So, divide Value (your total score for each story) with Cost (measured in story points) and sort your backlog by the result: the higher the value the more value I get back on the effort put in:


The above table is the resulting suggested prioritization based on the Theme Scoring technique and on calculating value/cost. Obviously it is worth noting that this is a suggestion only. It is an aid that I can use when prioritizing my backlog, it’s not something that replaces my thought-process – I still need to think and consider every item and e.g. be careful that I don’t miss any dependencies between stories (yes I know, we shouldn’t have any dependencies between stories, but sometimes we do anyway! :-)).

As always, I’m interested in input on how other people do things…!

Wednesday, July 8, 2009

Technical debt in Scrum projects

Code deteriorates with age. The older the system, the worse the maintenance. Why is this?

One answer may be that it is because of the maintenance itself: the things done over the years to keep the system afloat and the new features added, removed and changed. And people come and go. Some quit the team and new ones join. Sum it all up: deterioration.

Let’s talk about some of the causes and effects of deterioration.

Reinventing the wheel
An example of this is when a system contains several versions of the same function implemented at different times, often by different people, with only small differences. A similar situation is where a design (architecture) problem is solved in different ways in different places in the system. Either way, reinventing the wheel increases the complexity of the system and makes it harder to maintain. Developers won’t know which function is used where or even if they are still used at all (and consequently leave them there).

This is a negative spiral; the more complex the system gets the harder it is to understand, and consequently the harder it is to know if “my” problem has been solved already somewhere – which might lead me to unintentionally write my own version of a function that actually already exists. The lesson is to not reinvent the wheel. But it is easier said than done.

Maintainability is not something you can just add down the road. Building a maintainable codebase starts from the first line of code and it never ends.

Fear of changing what works – Legacy code
Once a system reaches a certain level of deterioration, people will be afraid to change certain parts. It’s often some critical and central component. Since it is central and critical it means it has been patched up numerous times, and changed and adjusted and changed again – over a long time. And hey, it works – now. And since it works, now, and since the code is so complicated and confusing, the developers don’t dare touch it. Consequently any features that would result in work in that critical component will be held back.

The way I see it, one of the root causes for this, is lack of regression testing abilities. With proper means for continuous and whole-covering regression testing, there is little to fear even when it comes to modifying old “legacy code”. Otherwise there are only two options when it comes to modifying legacy code; a complete rewrite, or just not touching it at all.

Lack of (efficient) regression testing abilities
The purpose of regression testing is to verify that every part of the system – even the less obvious parts – still works after a change or addition somewhere has been completed. The point with regression testing is to test a lot, over and over and over again. If you are in a situation where regression testing requires immense resources (which it does if you do it completely manually), then you will not be able or willing to do it as often or as extensive as is required. As a consequence your regression testing become inefficient, or even non-existent.

One of the steps in the right direction is to start automating your tests, and start running those automatic test cases continuously. The challenge, of course, is to;
(A) find a practical means - a tool - for automatic testing of your type of code, and
(B) figure out where to start.

I can’t help you with A (...psssst: JUnit, CPPUnit, CUnit, PHPUnit), but for B I suggest you just start somewhere. Don’t try to do it all at once – you won’t make it! Instead, just pick a simple starting-point, for example the most recent and newly added feature/module/component/part. Forget about the old stuff for now, just add automatic testing for the new things from now on. Something is better than nothing.

People joining and leaving
This is, unfortunately, unavoidable in most projects that are longer than just a few months. It happens. Either people get reassigned, or they choose to quit. And best-case, people join the team. Apart from a change in productivity caused by a team member leaving or joining, it obviously has an effect on the code itself too. People leaving will take knowledge with them, and people joining bring in new ideas (and misunderstand parts of what exists, too). One way to minimize the effects of this is to organize into “Feature Teams” that has a lot of close cooperation and joint commitments (like – tada! – Scrum suggests). This way you naturally spread knowledge among several people. It is also a pretty effective way of introducing new team members into the groove of things.

The classic method to minimize the problem of people leaving and joining is to write documents. I however argue that this is not the silver bullet for spreading & retaining knowledge. In fact, I think it's dangerous to rely on documents as the main tool for this; documentation is an extremely cost-inefficient and overrated way of transferring knowledge, and something that is often forgotten is the cost of keeping documentation up-to-date. As soon as documentation falls behind it becomes untrustworthy – and untrustworthy documentation will cause confusion and misunderstandings, and in the end no one will dare rely on documentation, and the time writing and updating it up to that point becomes waste.

In my opinion the guideline should be: don’t over-document – document just enough. Code Comments, I think, is a great benchmark of what level of documentation is “enough” for most situations. And remember; one excellent thing about code comments is that it is automatically (well…) kept up to date as the code changes – there’s little or no added cost for keeping it up to date.

Oh, and for the record, I’m not saying “Don’t write documents!”. If you really need to document then of course you should. I’m merely suggesting that you at the very least question the reason for doing that effort, and that you don’t forget to take into account the cost of keeping the document up-to-date as the system evolves.

Taking shortcuts - the Dirty that remains long after the Quick has been forgotten
“Well, we’ll do the quick-n-dirty fix now, just to get it done, and then we’ll go back later and clean it up...”. Have you ever said or heard something along those lines?

Short term gains such as reaching some immediate deadline makes it tempting sometimes to take shortcuts, and often it’s, sadly, a conscious decision. The problem with shortcuts is that they seldom or never get fixed afterwards, because there’s always that next deadline coming up with a new bunch of stuff to do with a new bunch of shortcuts that “has” to be made.

Doing things right from the beginning often requires a little more effort up front. And I think that different times call for different ways of acting. Sometimes it might be a correct decision to look only at short-term gains and cutting down on the immediate effort, and forgetting about those long-term consequences and drawbacks. But many teams and managers, I think, tend to be shortsighted by default – even when they don’t have to and there would in fact be room to do things properly. And in that case it is a matter of attitude (and competence). Do you do things fast and sloppy now and accept to pay for it later, or do you let it take a little longer now and reap the benefits of it later (for example in terms of costs saved on maintenance)?

It’s a challenge to figure out what is a shortcut and what is not. Remember Lean Software Development and the idea of “Extra effort” (and “Extra features”) being Waste. How do you know what is “Just enough effort”? There is no default answer to that. It depends. It’s up to you to figure that out for your system and for your business. But by figuring that out (or deciding on it) you will know what level to strive for; and anything below it is a shortcut and should not be accepted.

Don’t forget that you need to make sure that whatever your level is, it should be gut-felt by every team-member.

Bug fixes
Bug fixes have a tendency to deteriorate code. Enough patches in one place and the code will become more and more messy – at least if you have other problems too that cause code deterioration, such as people coming and leaving, inability to do regression testing, etc.

Bugs found in a production environment are often time critical to remedy. It can be tempting to take shortcuts to just fix the problem quickly and get a patch out there. But if you do that enough times but never take time to clean out the mess, you are destroying your system. See section above about taking shortcuts…

Summing things up – dealing with Technical Debt
A nice way to think of this deterioration of code is to think of it as “Technical Debt” - a term coined by Ward Cunningham.

Technical Debt is a long-term loan that we for some reason choose to, or have to, take from ourselves in order to achieve some short term gain. The Technical Debt increases for every individual loan and the debt never just goes away by itself. We have to pay “Interest” in the shape of things taking longer to complete because of this deteriorated code, and the only way we can decrease the Interest is by decreasing the loan – by paying “mortgages” e.g. by refactoring.

The most common (and worst) approach to dealing with Technical Debt is to ignore it. To pretend it doesn’t exist, and just push development forward without considering whether or not we’re taking loans.
The better approach to dealing with Technical Debt is to recognize it and have an active plan on how to deal with it in various situations.

I suggest you deal with technical debt into two steps: First stop increasing the debt further, and secondly start decreasing it. Only once you know that your debt is not increasing, you can start actively working with decreasing it.

To support you in the first step I suggest you insert a row in your Default Definition of Done that says “The Technical Debt has not increased”. It sounds trivial, but the intended effect is that it recognizes that it is OK that things take longer to complete if done in a way that doesn’t increase the debt - e.g. that it is OK to not take shortcuts. Remember to constantly remind people about this, and really do put your the money where your mouth is. Whenever faced with a choice, make sure you consider that the decision should be in-line with the attitude of letting things take longer in order to not increase the technical debt.

Once you’ve gotten used to that approach (it probably takes you a while and, if nothing else, will probably cause your velocity to drop significantly at first), the next step is to change the Definition of Done to instead say “The Technical Debt has decreased”. This is intended to recognize the fact that it is OK to also do some refactoring of things surrounding the current implementation “while you’re at it”. For example, urge all developers to modify the methods above and below the one currently worked in - even though it wouldn’t be necessary to complete the story itself! This way, for every new story completed the existing debt will decrease.
This type of refactoring will puts a lot of demands on your regression testing abilities. If you don't already have an automatic testing environment I suggest you start with introducing that first. Refactoring working code will (as explained in a section above) will otherwise be much too scary.

That's it from me for now. As always I'm interested in hearing other people's experiences and opinions in this matter.

Friday, July 3, 2009

Scrum Practitioners South of Sweden on LinkedIn

I just started a LinkedIn group: Scrum Practitioners South of Sweden.

The idea is to collect a bunch of people in the south of Sweden who want to meet face to face regularly, in an informal manner and share experience of Scrum, Agile and related topics with the intent of helping each other out and improving how we work.

Not sure if such a network already exists but I certainly see a need for it personally.

Follow this link if you're interested in joining.

Friday, June 26, 2009

Done Environment

In addition to intelligent version control, I like to have a set of “test environments” that map closely to my Agile Version Control policy.

First, it is necessary to have some sort of “Development Environment” where the most recent code from the Development Branch in the version control system can be tested and worked on.

Similarly, I like to have a “Done Environment” (sometimes also known as “Next Release Environment”, or “NR Environment”) where the most recent code from the Done Branch can be seen. The power of this environment is that it allows (forces) you to integration test early. If your Done Environment is an accurate representation (e.g. replica) of your live environment then you will be able to integration- and system test as early as the first story being delivered there – and I always include in the Default Definition of Done that the story should be working and available for demo in the Done Environment. In fact, I like to setup an automatic build and distribution script here, that every night packages the most recent version of the Done Branch in the version control system and puts it on the Done Environment. This way you know that you will always be able to see the stories that are Done-Done on in this environment, without any risk of them being infected by code in progress. This also means that once the release day comes you have (hopefully) already done a lot of the steps required, and consequently making a release becomes less of a deal than it otherwise often is; avoiding or at least minimizing the Big Bang problem.

If you’re short on servers, server-space and/or money (actually, regardless) I think it is a great idea to use virtual servers for hosting the environments. It gives a great flexibility when it comes to setting up new environments and cloning existing ones. If you have a live environment consisting of 40 physical servers, and you want to replicate that environment as a Done Environment, it’s not always practical to buy 40 new physical servers. Especially not if you have three teams and each has its own Done Branch, and consequently needs its own Done Environment (3 x 40 servers ;-)). But maybe it is OK to buy just a few physical servers, and instead run the Done Environment servers virtually. Virtual servers is a great and powerful tool – try it. You can find a list of software here: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines

Monday, June 22, 2009

The velocity log - another Scrum tool

Do you measure velocity? If so, how do you record it? (And if not: start!)

You measure (or should be measuring) velocity in order to be able to adjust the intake of future sprints, allowing the team to optimize the size of its commitment, facilitating for keeping a constant pace. What you should be measuring, at the very least, is the number of story points completed according to the Definition of Done in each sprint. This is, as you probably know, called Actual Velocity. You should also measure how many story points the team originally thought they would complete when they started the sprint (on the sprint planning meeting). This is called the Estimated Velocity.

I like to do the counting as part of the Retrospection. The team review what they have delivered and we do the math and record it.

The actual (and estimated) velocity are dependent on the context in which the team work; plenty of disturbances, impediments, new people etc will yield in a lower actual velocity. The velocity is also dependent on the sprint duration and on the team size. You want to keep both as constant as possible, so that the team really has a continuous pace – a “beat”. However, there’s a reality, where people have vacations and there are parental leave and educations, and there are holidays and day-before-holidays and what not. Despite best efforts, the sprint duration and the team size will vary over time – a lot, sometimes. This makes it hard to compare the velocities of different sprints; one sprint resulted in 25 SP and another 40. Was that because the environment was such that it allowed the team to complete almost double the amount, or was it because the sprint had a few extra days and a couple of people less on vacation?

I like to remove sprint duration and team size from the equation so I get something comparable over time (across sprints); because variances in velocity should point out differences in the context, and not differences in the sprint duration or team size.

For this reason, I record also sprint duration (in number of working days) and team size (in number of people), together with every pair of velocity (actual + estimated). I try to keep the numbers to integers, but sometimes I have to count “half” people (because of e.g. parental part-time leave, educations, or similar). I try not to go into more details than halfs – if I’m just consistent in the way I measure and in the level of detail, it won’t matter in the end – so why waste time and effort trying to figure out if I have 5.18 persons or 5. Anyway, with this information (Sprint Duration and Team Size) I can calculate the number of available man-days in the sprint (number of days x number of people), and then I can use that to calculate the “Estimated Story Points per Man-day” and “Actual Story Points per Man-day”. It will say something like “1.5”. It means that for every man-day the team completes 1.5 Story Points.

I’ve made an Excel (I always do ;-)) for recording velocity like this. I call it “Velocity Log”. It also has some nifty charts to plot velocity and trends over time. It’s available for download here:
VelocityLog-v1.xls

As an example, here’s the velocity of my current team over a couple of sprints:

Let me know your thoughts.

Wednesday, June 17, 2009

Why estimate using Story points?

The other day I was talking about why bothering using Story Points instead of just estimating everything in hours. Here's a link to the slides.

Traditionally, the aim has been to try to estimate time by looking intensely at a problem (or a task, activity, requirement,…) – so traditionally there’s a lot of effort spent on breakdown, analysis & estimation. But at the same time, we all know that estimating time is very hard and often pretty inaccurate – because software development is research-oriented and there are things that surface only once we start digging in, which then change the picture. We all know this – yet those time estimates tend to become commitments down the line.

I argue that it’s not only the problem (task, activity, requirement,…) that is hard to foretell; the environment where we solve the problem also has a significant effect on the amount of time it will take to complete. And often the environment is much harder to foretell than the problem itself. So, in the end it doesn’t matter how long time you spend analyzing, you will still have a large unknown in the shape of the environment.

The point I want to make is that estimating in Story Points is about recognizing that Time is a result of the Size of a problem solved within a certain Context.
Time = Size x Context

Story Points is a unit for representing Size. This way Context becomes something we can grasp by comparing Size to Time – which is why we measure Velocity.

Monday, June 8, 2009

Agile Version Control - another magic bullet

I have previously posted articles about what the magic bullet of agile software development is; to achieve flexible scope. As you may recall, the article talks about the three most important aspects of your Scrum implementation which make up the fundament for being able to achieve this scope flexibility:
  • Strict prioritization (having it and following it)
  • Definition of Done (having one and following it)
  • Story breakdown & formulation skills (understanding its importance, and constantly trying to improve)
In hindsight, I feel I have left out one important success factor when it comes to reaching a truly flexible scope; version control.

Most of us use one form of version control or another, but in my experience it is often (mistakenly so) only a matter of choosing the tool. The options are many, and range from “open source” to “incredibly expensive”. Once you decide on the tool, you use it for version tracking your codebase by checking out, committing and merging files. Perhaps you use branches in order to separate some work from other. But you may be missing an important tool; an Agile Version Control Policy.

If you do Scrum or some other form of Agile Development, you want to be able to actually finish your product increment every iteration, right? The purpose is to be able to achieve a flexible scope so that you can potentially ship on any day, at any given time. And what I am trying to get to here is that it doesn’t matter how well you succeed with your adherence to a Definition of Done if you don’t combine it with applying an Agile Version Control policy (completely regardless of which version control system you’re using). If you don’t have an Agile Version Control policy in place you risk mixing up your “Done-Done” code (code and assets related to User Stories that have been completely done according to the Definition of Done) with code which is in progress.

And guess what: if you mix Done code up with Code in progress, you’re no longer able to release at any given time!

Without Agile Version Control, shipping your software will mean that you first have to perform a series of tasks related to isolating code that is Done-Done from code that is in progress at that moment, and also often tasks related to doing last-minute polishing “while you’re at it”. The result is that shipping code is hard and a “big deal” and something that takes a lot of time. And what else does it cause? Yeah: crunch.

So how do you achieve this separation of Done-Done code from code in progress? Well, there are plenty of good articles out there to read: this one for example, written by Henrik Kniberg.

In short, what you need to do is dedicate a branch for containing that Done-Done code (let’s call this branch “Done Branch”), and define some rules for it. For example "all code on the Done Branch must adhere to the Definition of Done", "no code may be published to Done Branch without being reviewed", "code may only be published there by the branch owner", and so on.

Simplicity ftw! Here is a very simple Agile Version Control policy that I’ve been using;

Live Branch
Description: Create a branch from the trunk when you make a public/final release, that you will maintain for a while (until next release). Maintenance (patching) happens (for sake of simplicity) directly against this branch.

Branch policy:
  • All code in Live Branch must be tested and working.
  • All code in Live Branch must compile & build. In short: all code in Live Branch must follow Definition of Done.
  • Code may only be published to Live Branch by branch owner (who may delegate it).
  • Patches on Live Branch are immediately published to Trunk as the last step, and any conflicts are resolved on Trunk until code adheres to the branch policy. Note that you don’t resolve the conflicts on the Live Branch because it would imply updating Live with whatever is on Trunk (which can be a lot of changes).

Done Branch (which happens to be the Trunk, aka Main)
Description: Use the trunk (aka main branch) as “Done branch”. This will contain only code from stories that are Done-Done.

Branch policy:
  • All code in Done Branch must be tested and working.
  • All code in Done Branch must compile & build. In short: all code in Done Branch must follow Definition of Done.
  • Code may only be published to Done Branch by branch owner (who may delegate it).

Development Branch
Description: Create a branch from the Trunk in which the team works during the sprints. The branch exists across several sprints.

Branch policy:
  • All code in Development Branch must compile & build.
  • The Development Branch should be updated from Trunk often (suggest daily).
  • Conflicts with Trunk are resolved in Development Branch until code adheres to the branch policy.
Remember to update your code often from Development Branch (i.e. spark merges as often as possible!).

This simple policy relies only on three branch types; Live, Done and Development. Your Trunk (aka Main) is your Done Branch. There is a drawback with this policy though; if you have multiple teams who are developing (and finishing) stories which are supposed to be able to release independent of each other, then you will not be able to use the policy above (because Done-Done code from the two teams will be mixed up in the Done Branch). But, do you really have the requirement of teams to be able to release code independent of each other at different dates? Think carefully about this – do you really? If you do, fine: in that case you have to have a slightly more sophisticated setup (with a separate Done Branch + a separate Development Branch per team, and consequently you can’t use the Trunk as your Done Branch). You can read more about such a setup (Agile Version Control in a multi-team environment) in the paper by Kniberg that I referenced above.

As always, I’d be glad to hear about other peoples’ experience in this area. Please leave comments.

Thursday, May 28, 2009

When Scrum meets traditional quality assurance

The contents of this post has been growing on me for some time. It’s about the challenge of working with Scrum in an organization where there’s a lot of traditional (eg waterfall based) ways of doing things.

Let’s say you’re the manager of a team that develops and maintains some software system. Let’s say you’ve successfully introduced Scrum in that development team and they’re doing well and things are progressing. But, you don’t have any “Testers” on your team. Yet. Whatever testing is done is performed by the programmers themselves, in a more or less ad hoc way. Now, there’s a whole “QA Department” in your company, with resource(s) available for you. But it is an entirely different department that is not under your control. All you can do is request and allocate one or more resources from QA – but you’re not in a position to affect how they work. They are used to working according to a waterfall style project model. The processes that they follow rely on a “requirement specification” to exists ahead of time, before the implementation begins, together with a detailed and approved UI design ahead of time, a detailed and approved system design ahead of time, and so on. So the Testers from the QA Department will not be able to assist with any testing unless you have these things in place for them to start building test cases and test specifications on. What do you do?

One of the most central challenges for me right now with regard to Scrum & Agile Development is exactly that: How to continue working with Scrum in my development team and still manage to cooperate and integrate smoothly with another department that is not yet working according to Scrum or the Agile philosophy.

Is it possible to follow agile principles in only one part of the software development process?

So far, since I haven’t had any testers on my team, I have more or less been ignoring the problem and simply done things cowboy-style. I’ve run the project as a pure Scrum project, with user stories in a Product Backlog, with a Scrum Product Owner, with velocity measurement, etc, etc. And – in line with Agile – we’ve worked in a strictly prioritized order, so consequently we haven’t spent much time yet on those lower priority backlog items, and they would certainly not yet be possible to use for formulating any “Test Cases” in the QA Department’s meaning of the word. But the time is coming where my Scrum team and their way of working will meet the QA Department and its way of working. Will the two fit?

Don’t misunderstand me! I’m really looking forward to getting those QA resources on the team, and I’m 100% confident, regardless of philosophies and project models, that quality will benefit from the addition of QA resources – no question about it. But I’m worried that the benefit may be at the expense of “quality” in the development process itself (in the sense of me having to “sacrifice” certain Agile cornerstones).

In my view, “testing” and “quality assurance” is not something that is applied on top of programming as a separate stage. Quality assurance is something that is intertwined with the entire software lifecycle; from idea, through analysis, design and development, to integration and system testing, rollout, handover and maintenance. And since it is all intertwined inside the software lifecycle, it is not possible to limit Agile philosophies to programming only. To be precise, and to answer my own question above; No I don’t think you can be “agile” in an environment where you finalize a “Requirement Specification” at the beginning of the project, and then base everything on it; plans, designs, test cases, etc.

So, my plan is that together with the coming QA resource we will try to find a way for us to meet half-way; to find a way of working that requires minimal change in the QA processes and still minimal change in the Scrum implementation. I really think this can be done.

Here’s the approach so far;

What does the QA Department do? Well, they test software. What information do they need in order to be able to test? Well, they need a test plan, and they need a set of test cases. And what information is needed in order to create the test plan? A schedule, probably. Great! I can deliver a schedule, no problem. I just have a slightly different way of creating it than traditional projects.
And what information is needed in order to formulate test cases? “Requirements”! Could I deliver the backlog as “the requirements”? I suspect no. Darn. Why? Well, the User Stories alone are probably not detailed enough for QA to base test cases on; the level will be too high and there will be too much room for ambiguity, especially before we start working on the story. However, once we’re done with each story we’ll know more about the requirements. I don’t want to have to deliver the requirements ahead of time.

But wait. Didn’t I say that QA is an integral part of the software lifecycle? Yes I did :-). So what is Scrum? Iterative and Incremental. Can’t the “Requirement Specification” also be created Iteratively and Incrementally? And consequently; can’t the set of Test Cases (the Test Specification) also be created Iteratively and Incrementally?

So, the goal for us now is to make incremental deliveries of the “Requirement Specification” and of the “Test Specification”, in parallel to the incremental deliveries of software – sprint by sprint. We’re just getting started and are working out the practical bits in order to achieve this. For example, we’ll include in our Default Definition of Done that Requirements should be formulated. Secondly, we’ll keep track of which Stories are Done-Done (meet Definition of Done) at the end of each Sprint. The tester will use that information as input for his work. So at the end of each sprint there’s a new set of finished Stories accompanied with Requirements, that the tester can use as input to create a Test Specification increment. This way, the formulation of test cases is done one sprint behind the development. It might not be optimal, but considering that the tester is not full-time allocated to the development team, doesn’t sit with us, and that I’m not really attempting to reshape the entire QA department’s process all at once at this point :-), I think it’s a fair construction and a good trade-off.

So. That’s where I am right now. I’ll post an update as we progress!

I’d be glad to hear about other people’s experience about this type of situation.

Wednesday, April 22, 2009

How to adopt Continuous Refactoring?

This is a question that I have been asking myself, as well as received from others, a bunch of times. I am posting it here in the blog since I don’t have a ready-made answer. Please leave some comments on this if you have suggestions.

To clarify the question, let me begin by setting the scene for you;

Let’s say you are introducing Scrum. The team is, as always, a bunch of developers with varying experience, competence, and ages. You have read my blog ;-) as well as other even better sources of Scrum & Agile ideas and opinions, and you understand that the concept of “continuous refactoring” is central. You realize that anything is less likely to succeed if you try to solve everything at once in one single shoot; so it is better to work iteratively and deliver in increments, in order of priority. How do you get this message through to the developers, so that they really adopt it, wholeheartedly, in practice?

Because the problem here, in my experience, is that most people take a lot of pride in their work and “continuous refactoring” is easy to mistake for “rushing things out”. Some people just can’t get used to the idea of not attempting to do it “perfect” or “complete” from the beginning, and thereby find it extremely hard to deliver something that “just works” and is “good enough” for that particular product increment. Obviously, people are different (and so are programmers ;-)), so some will easily adopt this way of thinking, while it will be much harder for others.

How do you do it? It probably has much more to do with psychology and leadership than with Scrum per se, but I still think it is a relevant topic to bring up within the boundaries of this blog.

My suggestion on how to go about this is to educate. Talk about it. Have trainings. I have had lengthy discussions about this topic in the Scrum training sessions that I’ve held, and I can’t say that it has magically solved anything just like that, but I think the key to any change is for people to understand the need for it – and the first step for that is to talk about the problems surrounding BDUF (Big Design Up-Front, which is sort of the opposite of Continuous Refactoring).

Please leave some comments.

Tuesday, March 17, 2009

New employer - new Scrum & Agile challenges

Unfortunately I haven't had much time to write here lately. My excuse has to be that I just started a new job with an employer a little bit closer to home. I'm very excited about this opportunity, as the company I have now joined is more established in terms of its organization, practices and processes. First time really that I work for a company that isn't in the startup phase.

I'm just getting to know the people now, and studying the development and project models, so no new thoughts about Scrum or Agile just yet. Really looking forward to experiencing development practices in a mature organization (that supposedly have already adopted Scrum, in some way).

I'll be sure to post thoughts and experiences about this new Scrum & Agile journey, as I go.

Monday, February 16, 2009

Forget Change Management - it undermines your chances of success. All changes should be free!

When you commit yourself to delivering a certain set of requirements ("requirement specification", "statement of work" or whatever you call it) by a certain deadline you automatically create a need of ways to manage change – because change brings risk and disorder, and it risks ruining your time-plans that you have spent so much time figuring out. It risks ruining the architecture that your designers & architects spent so much time thinking through, and it risks turning those already completed programming & testing efforts into waste because a feature is made obsolete.

What's even worse, if you're the project manager then you're likely to be the lucky person stuck with the bill and you have to work out who's going to pay: your company or the customer?
It can be a nerve-wrecking job to be the one that have to make that call to the customer and explain that "this recent request will end up costing 5.000 EUR more and it will delay the project with 3 weeks". And if your requirement specification is the least bit vague or ambiguous at this point, you might end up in an irritated argument or negotiation with the customer about him considering the feature part of the “original scope”, whereas you consider it a Change Request. Not really an optimal situation. But under such circumstances of course you will require a clear way of managing these ugly changes.

So what can you do to minimize these risky aspects of software project management? The traditional way to go is to become even more thorough the next time during Requirements Analysis. I.e. you put even more focus on planning and on trying to figure out even more of the details up front. You have longer pre-studies. You do bigger and more detailed design up front. This way you'll be able to give yourself a better picture of the project before you sign off the requirements and set any deadline. In essence, you build your walls higher and thicker around you. You strengthen the foundation for your Change Management Process and thereby make your project less likely to be exposed to Change and the risks they bring.

With the above reasoning change is something ugly that you want to avoid. But wait. Is change really a bad thing?

I argue that change is great. Change is fantastic.

Screw Change Management processes and the administration they bring!
Yes, I said it. In fact, I think we need to give the customer all changes for free, always!

Change means the customer has gotten a different or stronger opinion than he had before, and if I don't adapt to it then the customer will be less happy than otherwise. And what is our number 1 objective? …No, silly, not to make money. Our number 1 objective is to make the customer happy - because happy customers are returning customers. And as a consequence we will make money.

The problem here is much larger than the Change Management process alone. The problem is in the entire traditional approach to software project management and software development.

With a traditional methodology you will definately need Change Management - no doubt about it. There are many great project managers and companies out there, I'm sure, who are working with traditional methods and who are excellent at Change Management.

But I think remarkably many of those never thought twice about the software development methodology they adopted. They just chose one that everyone else was using (for that reason: because everyone else was using it) instead of adopting a software development methodology that maximizes their chances of success. At the end of the day, if you're not NASA and you're not working against the FDA with tons of beurocracy and laws to adhere to, then you're much better off using an Agile development methodology.

You have to accept that you can’t foretell the future. It would be very nice if you could, but you can’t. Sorry. And consequently; change is inevitable – you can’t wish it away. You also have to realize that the customer doesn’t know what he wants – he discovers that as he goes along. Consequently; change is inevitable. Furthermore, you have to realize that developers don’t know how to build the software up front – they discover that as they go along Yes, consequently; change is inevitable.

You can sit there and sob and dislike it all you want, but it doesn’t change the facts.

This is important, and a huge difference to traditional assumptions, so I’ll recap:
  1. Unforeseeable sh*t happens that will affect your plans.
  2. The customer doesn’t know what he wants – he discovers that as he goes along.
  3. The developers don’t know how to build it – they discover that as they go along.
Most people will agree that the above is true. So, why do we rely on dusty old project management methodologies that assume the exact opposite??

The first thing you need to do is to stop committing to both scope and deadline up front. Commit to date rather than scope. This is sometimes called “date-driven development”. You achieve it by putting the requested features in one single list that is prioritized by the customer. Allow changes to priorities, as much and as often as the customer likes. But make sure the priorities are always kept unique for each item in the list and that the priorities are strictly followed, i.e. work is carried out in strict order of priority. As a consequence of adopting this approach there will be no such thing as an “original scope” – because that would imply that there was a “before” and an “after” some event (e.g. a “change”). “Feature creeping” is no longer a bad thing. As long as the customer sticks to the one single prioritized list of features he can have as many features “creep” in as he wants; he can replace all the items with others from one day to another. As long as the customer is happier in the end. With this approach there is no such thing as a “scope change” anymore: the scope instead is something that evolves over time by continuous readjustment .

Secondly, you need to stop making detailed plans. Plan for God’s sake, don’t misunderstand me, but just don’t overdo it. Forget about figuring out what exactly will be done, by who, when, before or after what, two-three months or more away from now. Try to ask yourself; are you creating the time-plan the way you WISH things will happen or as you KNOW they will happen? I’m guessing most of the time the answer will be that it’s the former. The effort spent on creating silly wishful-thinking time-plans is much better spent on something more constructive in terms of bringing the team closer to the end goal (a happy customer) – like getting coffee for the developers. Note that I am not telling you NOT to plan. I am just telling you that all plans are wrong and rarely survive more than one minute in contact with reality. So just minimize the amount of time you spend making them.

Thirdly, start involving the customer. Do it early on. And make sure you have as short feedback loops as you can. With a traditional approach, presenting the product to the customer is often scary because you fear there will be changes, and those changes can be ugly. But you have to remember that those changes are a necessity for the customer to become maximally happy. The problem is not the feedback, the problem is when you have a long lead-time between points of customer involvement (and consequently risk having put in a lot of effort into something that is wrong or will change). So, remember to keep feedback loops (very) short, from the very start of the project to the very end. I suggest 1-3 week loops. Notice that this puts a different demand on the amount of customer involvement than with a traditional approach: the customer needs to be involved (much) more, and earlier. If your customer is not comfortable with that then you’re better off with a traditional approach – but you will less likely to succeed, and the project will be much more expensive because you will have to put risk margins in your pricing.

Fourthly, derive a release plan from reality rather than dream one up ahead of time. Don’t lie to yourself or (worse) the customer by claiming you know exactly what you will be able to deliver by when. You don’t know this! And anyone who claims to is a liar. Instead, commit only to time. Then continuously observe your actual progress rate and use those observations to calculate what parts of the scope it is likely that you’ll be able to finish before the deadline date. You work in priority order and have short feedback loops involving the customer that spark “change” often and early. When you reach the deadline date you have the most important things done.

Fifthly, test early and continuously, and always finish things (completely). Don’t leave some parts of testing for later. Don’t postpone code reviewing. Don’t wait with code merges. The sooner you do things the easier and cheaper they are. Started, half-finished work has no value. For every new feature you start without finishing completely, you add to a pile of more or less known remaining tasks. Over time this pile grows to a mountain of unknown stuff that will get you in the end. Avoid this by really completing things early and do all types of testing early. Continuous integration is key here, and automated testing is a helpful tool.

Since change and unpredictability is inevitable, we have to find a way of working where change is not made ugly by processes. Where there is as little waste of effort as possible when change happens. This is what Agile Development is about – a software development philosophy which covers process frameworks such as Scrum, XP, FDD, DSDM and others. But remember: tThis isn’t the silver bullet that automatically solves all your problems, but it brings maximally good odds of succeeding. Software development and project management is still hard, but at least you’re not making it harder than necessary.

If you’re interested in reading more about Agile and Scrum here are some links:
My blog: http://scrumftw.blogspot.com/
Agile Manifesto: http://www.agilemanifesto.org/
Wikipedia about Agile: http://en.wikipedia.org/wiki/Agile_development

Thursday, February 5, 2009

Working smarter is better than working harder - How to increase velocity

"Why is progress slow? Why does everything have to take so freakin' long? You have to push harder! Work more."

This is something that I've heard a lot, especially from certain types of managers. Let’s look at this more closely, and what it means, and what it is an expression of.

I think it is reasonable to assume that it’s not a matter of the manager actually wanting people to work harder, faster or more just for the sake of doing so. No, instead I think (I might be naive ;-)) that it indicates the manager’s wish to have stuff done sooner. "Stuff done sooner", that is key here. Can this really be done without having people work harder/more? Yes! If you work smarter!

The time that something takes to complete (from its very start, to the time that it is ready) is called “Cycle Time”. You can learn a lot more about this if you study Lean Software Development, I won’t go much further into details here.

Let's investigate what factors affect Cycle Time. How “hard” people work is one factor, yes absolutely. At least up to the point where negative stress kicks in and its consequences appear; defects and demoralization, lacks in communication, loss of cooperation and team synergy effects, and many other ugly things.

At this point I could dive into Value Stream analysis, talk about Value Stream Maps and discuss how to apply the Lean Software Development tools supplied by Mary and Tom Poppendieck et al. But I won’t do that now. In this blog post I will limit myself to only the “implementation part” of the development process. You should know that there is a lot more room for shortening “Cycle Time” than what I will talk about below. With Lean and its tools you should go far beyond only the sprints and observe the steps before and after implementation and focus on the whole chain – the entire “value stream” that flows through your business.

So, back to the subject. In Scrum we continuously measure at least a part of the Cycle Time: We measure velocity. The velocity is the speed with which the team finish things, according to your Definition of Done. Velocity is measured in “Story Points”. You can observe it over a sprint (SP/sprint) or over man-days (SP/manday), or variations of this sort. The point is that you measure the velocity in a consistent way, and you record your findings (in something that I call “the Velocity Log”).

If you say you want to shorten your Cycle Time, i.e. shorten the time it takes to finish something, then what you really want to do is increase your velocity. The bitch here is that the velocity is not something that you can tune by itself. It’s not a knob that you crank up and voila stuff goes faster. The velocity is a result of the environment in which you work. By changing the work environment you will change the velocity. It doesn’t work the other way around. Sorry.

I think we generally need to be much more focused on “helping” the velocity up. It is easy to stare oneself blind on a deadline and all the stuff that has to be done by then, dig in and spend all energy on completing those stories. during such times I think we often forget, or at least seriously underestimate, the power of improving the work environment and the effect it would have on the velocity. This is what "Continuous Improvement" is all about.

Now, what does “work environment” mean?

Unplanned items. They happen all the time, during your sprint and are typically part of your work environment. These are the things which you for some reason just have to do, but that you really shouldn’t (because the team’s job is really to work on the sprint backlog). These are those “negative additions/changes” to your sprint. They affect your team’s ability to focus on the sprint commitment. What you want to do is record your unplanned stuff by putting a note in your “Unplanned”-section on the Scrum Board, for everyone to see. Then you need to work on removing the source of those unplanned items. In my opinion, unplanned items is the number 1 opportunity to increase velocity. Find the root-cause of every unplanned item, and consider if you can’t insert a task into your Impediment Backlog about removing that root-cause. Either way, it is the Scrum Master’s and the Scrum Product Owner’s number one priority to shield the team from these things so that the team is allowed to focus 100% on the sprint backlog.

Concentration breakers: Email. What can I say. As a communication tool it is great, but the way email has been integrated into our daily life is not. If you get a “You’ve got mail!” dialog every time you receive one then your concentration will be broken as many times a day as the number of emails you get. Some days I get hundreds of emails. Jeez! If I was a developer today I would consider not checking my email more than once or twice during the day, and have the client shut down in between. In a Scrum team we rely on face-to-face communication more than written stuff anyway, so believe me, the really important stuff will reach you anyway without you automatically checking your email every minute. And get used to the fact that if you want someone something, then you need to walk over to them and speak to them face to face instead of writing an email.

Concentration breakers: Skype. It is indeed a great tool. In our company we rely a lot on short-lived Skype group chats that we create for various purposes, and we use it a lot for internal communication. It’s a great way of communicating and cooperating. But it can also be a huge concentration breaker. Skype announces new messages by popping up a new window (in the background), with a system tray popup and/or by coloring the chat window taskbar button orange. So every time you receive a message (personal or in a group chat) your concentration will be broken. I guess you learn how to ignore this after a while, but I’m pretty sure that ability is individual. My advice: be careful if you’re using Skype. I suggest killing it altogether and rely on face-to-face communication instead. If you decide to go that route though, make sure it’s a team decision and not enforced as a policy or company rule.

Concentration breakers: Background noise. We work in an open office landscape with very few closed offices. Most of the people sit next to each other in that landscape in order to promote cooperation and communication. The clear drawback with an open office landscape is the noise level from people talking, writing, from computers buzzing and so on. If you’re a manager reading this: don’t be a cheap moron. Invest in sound absorbing cubicle dividers, even if they are more expensive than regular ones (or compared to not buying any dividers at all). If there’s one thing that directly affects the concentration of every single person in the open office landscape – all day, every day! – it is the background noise level. Yes, we can always ask everyone to talk quietly and step into a meeting room for those longer discussions. But also consider: how much are you crippling freedom of communication (and cooperation) by not buying those sound absorbing dividers? It is a low cost for ensuring optimal work environment! While on the subject: seriously consider installing roof- and wall-mounted sound absorbents too.

Unnecessary manual work. Stop and reflect: are there any tasks/actions that you perform often which could be partially or completely automated? Even if the effort of automating it is somewhat large (maybe several days or even weeks of work to build automation or refine tools), it might be worth it in the end. Is it only you that have to do this manual task every time? Maybe it is every developer. 5 minutes a day, for every developer in a 25 person organization, equals 45 man-hours every month. Consider automating many of those manual tasks, and optimize the efficiency of those that you can’t completely automate. If you think about it I’m sure you can come up with a bunch of things. Have a workshop. List stuff and prioritize. I’m sure your Scrum Product Owner will listen if you run the numbers. If you have tips on tasks that can be automated, please feel free and add them as comments to this blog post!

Technical debt. The higher technical debt you have the longer things will take. The system is more complex, harder to test, things are harder to implement, and so on. Addressing technical debt is a subject on its own, which I will come back to in later blog posts. For now I'll just advise you to keep in mind that it directly affects your velocity. You should do what you can to (at the very least) immediately stop increasing your technical debt even further, by letting it take longer to do things "right". It is a short term investment for a long term gain. The bad approach is to do it the quick and dirty way. Remember: the dirty remains long after the quick has been forgotten.


So, if you think progress is too slow, consider working smarter instead of harder. It has a much bigger impact on your cycle time than crunch and overtime has. I really hate crunch. And in so does everyone. Trying to continuously improve the work environment and thereby increasing velocity should be way up high on your agenda! That is how you get things finished sooner, without killing yourself and without sacrificing quality.

I would appriciate comments on your tips and tricks and experiences regarding how to increase velocity!

Friday, January 23, 2009

Updated backlog manager tool in Excel

I have updated the Backlog Manager tool now, it's a bit simplified compared to last time. Changes since previous version:

- It works in Microsoft Excel 2003 now (previously only worked in 2007).
- You may now freely delete and insert rows in the backlog without risking that the list gets screwed up (due to missing formulas and such).
- There are no hidden columns or weird formulas in the Excel.
- The Card Generator no longer uses the ID-column, but instead the Autofilter result, so you can now filter with full flexibility and the Card Generator will generate cards for the visible rows only.
- The New Story dialog is simplied and looks better :-)
- The index card template looks better and is somewhat simplified.
- Some redundant columns were removed (simplify ftw! :-))

I have updated the old post from September last year so it links to the new version of the Backlog Manager tool / Excel instead.

Anyway, you can find it here too:
BacklogManager-v8.xls

Good luck!

Thursday, January 22, 2009

User Stories from the trenches

I couldn’t resist the topic. But for the record, credits to Henrik Kniberg for the “…from the trenches”-bit :-).

The past few weeks have been hectic, and I haven’t really had the time to post anything new in a while. The reason is that I’ve been in a phase here where we’re ramping up a project and I have been heavily involved in trying to form an initial backlog with an initial set of reasonably meaningful User Stories for that project. I.e.: we’ve been involved in getting a (very large) project “vision” down into a set of stories.

Our traditional method for this was to have the Product Manager (Customer) produce a written description of each feature that he wanted, and then he would hand that over to the Development Department where we (“we” being the team and the Scrum Product Owner: as you may know we use the somewhat controversial “Product Owner Proxy” construction in our organization) would take ownership of that written “spec” and break it down into User Stories that were prioritized and later estimated and delivered in priority order.

My problem with that approach was (and is) that the Product Manager usually (for whatever reason) spent alot of time writing these “functionality specs”.
I have felt that this goes against my core attitude/rule-of-thumb “Never document more than that you have to ask to understand” and that it is pretty "un-scrum". I’ve been wanting to find a better and more Agile way of moving from “The vision inside the head of the Product Manager” to “Something that the team understands, which can be prioritized and brought into the Sprint Planning meetings”.

In this case, with this development project being such a large one (a new product, with so many functions), it would simply have been an overwhelming task for the Product Manager to rely on our traditioinal approach and write all those "function specs" up front – at least considering the usual level of detail in those spec documents.

My idea was to skip the “detail specification” step alltogether and jump straight from the vision to User Stories, and thereby save time by having the Product Manager produce User Stories directly which later could be prioritized and further broken down. This would somehow force the Product Manager to skip details and thus not try to think of everything up front. Oh, and of course, we’d be forced to rely much more on face-to-face communication between the team and the Product Manager, which would be great: “two people by a whiteboard” is always a many times more efficient method of transferring information than by document, right?

So that was the plan, and this is what I have been struggling with for the past weeks (plus celebrating Christmas and New Year :-)).

I can’t say that it went great maybe, but I'm still pretty satisfied with the result so far. Anyway, I wanted to take this opportunity to really reflect over how things went and try to gather my own thoughts about it and nail down some lessons.

Here are some of the challenges with the past few weeks;
  • I needed to find a way for the Product Manager to “empty his head” into something that could be used as basis for formulating some fundamental stories.
  • I needed the Product Manager to understand what User Stories are and why they should be formulated in a certain way, what level of detail they should have, and so on.
  • I needed to find a way – a method and tool – to document the User Stories which would fit nicely with the tools we use further down the road: we use the Backlog Manager tool that I have made available for download on this blog in a separate post.
I approached this by introducing MindJet MindManager to the Product Manager. I know there are open source and other free alternatives out there too, but I haven’t tried any of them myself.

I use MindManager for many different things, and I find it great for emptying my head down into symbols, boxes, notes and relations. So the idea was that it could be used by the Product Manager too, to get the rough set of functions and other thoughts and notes down into a visible map. That map could hopefully then be used to look at to formulate User Stories; e.g. by studying a subtree and all the nodes there to remember what needs to be covered by the backlog.

The map ended up looking something like this, but much bigger (the map below is just a very small dummy example, obviously):

In retrospect, we experienced a few major drawbacks with this method of using a mindmap to describe the vision of the project and have the Product Manager use it as basis for formulating stories;
  1. Coping with keeping the map up-to-date with the stories: as stories are formulated, iterated, thought about, discussed and reformulated, after a while there are changes in the perception of how things are supposed to work. We failed at keeping the map up-to-date throughout this process. After a while the list of user stories conveyed a different picture than the map. However, I think it might be ok that the map is a “consumable” that can’t be trusted after a while; it’s just one tool used along the way and is not really an artifact by itself.
  2. Ones stories were formulated, they ended up being too small. The initial backlog ended up containing much too many stories and each story gave just a too fragmented view of what should be delivered. The symptom of this was that it was very hard for the team to grasp the project as a whole, even with a lot of communication with the PM. The breakdown done was simply much too detailed and too early.
I think my main lesson here is to try to get the Product Manager to just summarize each desired feature as a Theme or Epic (see separate blog post), rather than in a whole bunch of small stories. If there are important details about the function that the Product Manager wants to mention he should write keywords as “notes“ to the story, and not try to write each as a separate story.

Also if there are things in the Epic/Theme which can be given different priorities, obviously those can be broken out as separate Themes/Epics. The ambition should however be to not break everything down in detail.
The finer breakdown, into “ready-ready” (see separate blog post) User Stories should be done by the team in cooperation with the Product Manager – not by the Product Manager himself.

So to summarize on the lessons learned so far:
  • Get the Product Manager to use a mindmap to empty his head into, with regard to a certain vision of function(s).
  • Get the Product Manager to look at the map and its subtrees and use that to write Themes (or Epics), formulated as User Stories (“As a I want to so that .”.
  • Get the Product Manager to write those Themes/Epics in the same or similar tool that you will be working with in Development (e.g. if you can, directly into an empty Backlog Manager excel, if that’s the format you’re using).
  • When the Product Manager feels he’s covered the most obvious and/or most important Themes/Epics, bring in the team. Let the team and the Product Manager together work on the finer breakdown – of the most important Themes/Epics.
  • Talk alot about the Themes/Epics: between Product Manager and team. Let the Product Manager describe each Theme, if you have an updated mindmap you can use that as basis for a presentation about each, as part of the breakdown process.
  • Break down the Themes/Epics one by one, and work face-to-face. Even if it takes the PM and the team a long time, I have a feeling that it is time saved in the end: because the team will get an excellent view of the backlog. A challenge here is to heavily moderate discussions about how to build the stuff: solutions shouldn’t be discussed at this point. That’s done when the stories enter the sprints.
  • Don’t attempt to break every Theme/Epic down into smaller stories from day one. Make sure they’re explained and moved into the product backlog, but break them down in priority order: start with the most important ones. Leave the lower priority Themes/Epics for later sessions. Just because they’re big it doesn’t mean that you can’t estimate them. Remember: rather roughly right than precisely wrong.
  • Come back to the bachy hklog after every sprint and review the Themes/Epics and let the team and the PM break them down once they surface in the backlog.
Here is an example of a subtree which could be used as basis for writing a Theme and later for discussing around as part of breaking it down into smaller stories;

Example of a Theme for the above subtree: "As a Administrator I want to be able to manage users so that I can control who has access to what."

Once the team and the Product Manager break it down together, they'd probably end up with something like the following user stories:
  • As an Administrator I want to be able to add new users to the system so that each user can have his own personal account to login with.
  • As an Administrator I want to be able to search for users and and be able to sort the resulting list in various ways so that it maximally easy to find what I'm looking for.
  • As an Administrator I want to be able to Clone an existing user so that I create a new one with the identical data pre-filled so that I don't have to retype everything.
  • ...and so on.
We’re still not done here and have some distance to go before we can say that the backlog is in an “OK” shape. I’ll revise this post and/or make additional posts with more thoughts and experiences as we go along.

Oh, by he way. I always appreciate feedback about your experiences. If you have any clever ideas or thoughts about this or how you work with it in your organization, please leave me some comments.