Combine Scrum with XP

How To Combine Scrum With XP
To say that Scrum and XP can be fruitfully combined is not a controversial statement. The difference is that while Scrum focuses on management and organization practices, XP focues mostly on actual programming practices. That's why they work well together - they address different areas and complement each other.

Pair Programming
Some conclusions about pair programming:
 * Pair programming does improve code quality.
 * Pair programming does improve team focus (e.g. when the guy behind us says "hey is that stuff really necessary for this sprint?").
 * Surprisingly many developers that are strongly against pair programming actually haven't tried it, and quickly learn to like once they do try it.
 * Pair programming is exhaustive and should not be done all day.
 * Shifting pairs frequently is good.
 * Pair programming does improve knowledge spread within the group. Surpriningly fast too.
 * Some people just aren't comfortable with pair programming. Don't throw out an excellent programmer just because he isn't comfortable with pair programming.
 * Code review is an OK alternative to pair programming.
 * The "navigator" (the guy not using the keyboard) should have a computer of his own as well. Not for development, but for doing little spikes when necessary, browsing documentation when the "driver" (the guy at the keyboard) gets stuck, etc.
 * Don't force pair programming upon people. Encourage people and provide the right tools but let them experiment with it at their own pace.

Test-driven Development (TDD)
This, to me, is more important than both Scrum and XP. Here's a 10 second summary of TDD:

//Test-driven development means that you write an automated test, then you write just enough code to make that one test pass, then you refactor the code primarily to improve readability and remove duplicaiton. Rinse and Repeat.//

Some reflections on test-driven development:
 * TDD is hard. It takes a while for a programmer to get it. In fact, in many cases it doesn't really matter how much you tech and coach and demonstrate - in many cases the only way for a programmer to get it is to have him pair program with somebody else who is good at TDD. Once a prigrammer does get it, however, he will usually be severely infected and wil never want to work in any other way.
 * TDD has a profoundly positive effect on system design.
 * It takes time to get TDD up and running effectively in a new product, especially black-box integration tests, but the return on investment is fast.
 * Make sure you invest the time necessary to make it easy to write tests. This means getting the right tools, educating people, providing the right utility classes or base classes, etc.

We use the following tools for test-driven development:
 * jUnit/httpUnit/jWebUnit. We are considering TestNG and Selenium.
 * HSQLDB as an embedded in-memory DB for testing purposes.
 * Jetty as an embedded in-memory web container for testing purposes.
 * Cobertura for test coverage metrics.
 * Sprint framework for wiring up different types of test fixtures (with mocks, without mocks, with external database, with in-memory database, etc).

In our most sophisticated products (from a TDD perspective) we have automated black-box acceptance tests. These tests start up the whole system in memory, including databases and webservers, and access the system using only its public interfaces (for example HTTP).

This makes for extremely fast develop-build-test cycles. This also acts as a safety net, giving the developers confidence enough to refactor often, which means the design stays clean and simple even as the system grows.


 * TDD on new code:** We do TDD for all new development, even if that means initial project setup takes longer (since we need more tools and support for test harnesses etc). The benefits are so great that there really is no excuse not to do TDD.


 * TDD on old code:** TDD is hard, but trying to do TDD on a code base that wasn't built using TDD from start, that's really hard.


 * Lesson learned:** If you are stuck with having to do manual regression testing, and want to automate this away, don't (unless it is really easy). Instead, build stuff that makes manual regression testing easier. Then consider automating the actual testing.

Incremental Design
This means keeping the design simple from start and continuously improving it, rather than trying to get it all right from the start and then freezing it.

Spend a reasonable amount of time refactoring and improving existing design, and rarely spend time doing big up-front designs. Continuous design improvement is mostly an automatic side effect of doing TDD.

Continuous Integration
Most of our products have a farly sophisticated continuous integration setup based on Maven and QuickBuild. This is extremely valuable and time-saving.

Our continuous build server acts as the "judge" or reference point from which to determine the health of all our codebase. Every time someone checks something in to the version control system the continuous build server will wake up, build everything from scratch on a shared server, and run all the tests. If anything goes wrong, it will send an email notifying the entire team that the build failed, including info about exactly which code change broke the build, link to test reports, etc.

Every night the continuous build server will rebuild the product from scratch and publish binaries (ears, wars, etc), documentation, test reports, test coverage reports, dependency reports, etc, to our internal documentation portal. Some products will also be automatically deployed to a test environment.

Setting this up was a lot of work, but worth every minute.

Collective Code Ownership
Encourage collective code ownership. Pair programming with frequent rotation of pairs automatically leads to a high level of collective code ownership. Teams with a high level of collective code ownership have proved to be very robust, e.g. their sprint doesn't die just because some key person is sick.

Informative Workspace
Let the team have access to whiteboards and empty wall space and make a good use of this. The biggest problem is old junk accumulating on the walls.

Coding Standard
Implementing a coding standard is very useful. It takes almost no time at all, just start simple and let it grow. Only write down stuff that isn't obvious to everyone and link to existing material whenever possible.

Most programmers have their own distinct coding style. Little details like how they handle exceptions, how they comment code, when they return null, etc. In some cases the difference doesn't matter, in other cases it can lead to a severely inconsistent system design and hard-to-read code. A code standard is very useful here, as long s you focus on the stuff that matters.

Here are some examples from our code standard:
 * You may break any of these rules, but make sure there is a good reason and document it.
 * Use the sun code conventions by default: http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html
 * Never, ever, ever catch exception without logging the stack trace or rethrowing. log.debug is fine, just don't lose the stack trace.
 * Use setter-based dependency injection to decouple classes from each other (except of course when tight coupling is desirable).
 * Avoid abbreviations. Well-known abbreviations such as DAO are fine.
 * Methods that return Collections or arrays should not return null. Return empty collections and arrays instead of null.

Sustasinable Pace / Energized Work
Many books on agile software development claims that extended overtime is counterproductive in software development.

About a year ago one of our teams (the biggest team) was working insane amounts of overtime. The quality of the existing code base was dismal and they had to spend most of their time firefighting. The test team (which was also doing overtime) didn’t have a chance to do any serious quality assurance. Our users were angry and the tabloids were eating us alive.

After a few months we had managed to lower people’s work hours to decent levels. People worked normal hours (except during project crunches sometimes). And, surprise, productivity and quality improved noticeably.