Executing all SQL files in a folder

SET dbName=RemmicomMeetingDb
SET sqlCmdCommand=sqlcmd -S . -E -i

%sqlCmdCommand% "DropAndCreateDb.sql"

FOR /F %%i IN ('dir /b /on *.sql') DO %sqlCmdCommand% %%i -d %dbName%

Incremental Software Delivery

What i see occur quite often is that User Stories are to big, resulting in work that is not done at the end of the sprint.

Lets take a look at the story below:

As an office manager I would like to be able to lookup contacts quickly so that I can quickly find someone's contact details.

Acceptance criteria

• Given an Office Manager enters "como" When I press the search button Then he/she should see the contact details of everyone who's full name contains "como"

• Given an Office Manager enters"ce" When I press the search button Then he/she should see the contact details off everyone's title containing "ce"
• ...

Now if we think about this for a second, we could start to break down the tasks as follows:

• Create the database tables
• Create the data access layer
• Create the DTO's
• Create the UI
• Create the service layer

If we stop and think about this a second it does not really make sense in a incremental delivery mindset. If you reflect on this from a Form Follows Function Perspective, that the shape of an object should be based upon its purpose. This is a faulty breakup of the feature we are trying to build, since we are going to make the purpose - storing the data in a database and then transfer it to a screen - take precedence over the function, searching contact details.
It makes more sense tobreak up the screen in functional components as shown below:

Then we could break the feature up as follows:

• Display Search Results
• Enter Search Criteria
• Display status message "x out of Y contacts"

This will require developers to shift there mind from "Horizontal thinking" - thinking in terms of technical layers, to vertical thinking - what I often refer to as "Slicing the Cake". This enables you to incremental delivery of features and get earlier feedback.
The most important thing is the green grid displaying the search results. From a technical perspective this seems a bit weird, since we autmatically think that its not usable once we have hundreds of contacts. But this mindset is wrong, it is workable, whether its practical that is another thing.
If we would take this approach we could deliver the following screen to the Product Owner:

If he can get his hands on this he can allready take a look and maybe he could allready experience this bit of functionality and conclude he would like to add some fields to the grid like email and phone, that he did not think of immediately.
So an extra task is added to the story "Add email and phone fields to the grid"
In the meanwhile wile the product owner is allready playing around with the screen we can allready add the yellow part to the screen:

This will enable very short continuous feedback loops, moreover working like this will help you deliver functionality sprint after sprint.
This is what the end result could look like

As you see apparently there was no more time for the status message at the bottom of the screen and this task has been removed from the story another change from the initial version is also the "Search" button.
Find here an overview of what was implemented and what was removed:
As an office manager I would like to lookup contacts quickly so that I can quickly find someone's contact details.

1. Display search results: Done

1. Enter Search Criteria: Done

1. Display status message "x out y contacts": removed

1. Add phone and email fields to the grid: Done

1. Filter results grid when typing text: Done

1. Remove search button: Done

There is one caveat with this approach, business user tend to like alot to add stuff, but they really dislike removing stuff… But this is where the timebox principle kicks in, if for instance there would be no more time left than the 2 last tasks (tasks 5 and 6) than the Product Owner might still choose to accept the story as done and the Functionality could potentially be shipped.

Scrum.org PSM II Certification

I reached a major milestone and obtained my Professional Scrum Master Certification II last week. A put alot of effort in it and it paid of.

Something on Continuous Integration

In the past I have been experiencing some weird practices concerning CI. In the following article I will outline some do's and don'ts when implementing continuous integration.

First and foremost it is ok to break a build that is why the thing is there you know. Comparing the number of broken builds against the ones that don't break is not a good measure of quality instead measure the time to fix the build.

Do Integrate static code analysis in your build like FxCop, Ndepend and Simian. Giving you early feedback on your code quality. If the tools give you warnings solve them as well. If you keep up on regular basis this will keep your code base clean.

Do Integrate your Unit tests to run every build. Again providing early feedback on integration issues.

Do provide visual feedback (Andon) that give people an immediate feedback when a build fails.

Don't force developers to do a get latest, run all their unit tests and then check in. This is a waste of time. But when a build fails fix it!

Don't seek who broke the build. Something that is very common is that people are very eager to investigate who broke the build and pointing fingers. Instead look what broke the build try to fix it, this will enhance collective code ownership.

Don't check in, close your laptop and sprint out of the office. This practice kills team morale.

Determine Iteration Length

Something I come across quite often is that teams don't consciously determine the iteration length of their sprints. They usually take 2 weeks as some default length. This is not really the way to go… Lets see what criteria we can put forward to fine tune the iteration length.
Here are the 3 major factors to consider:

• Project Duration
• End-User commitment (how available are they to provide feedback and input)
• Maturity of the team

The shorter the project the shorter the sprints enabling to have more feedback cycles, minimizing the risk building the wrong thing. This however will require a significant amount of commitment of the stakeholders (weekly review meetings, availability to clarify and develop the product backlog with the Product Owner,...). Short sprints also have one big advantage they are more prone to fail… The reason why I see this as an advantage is that problems will surface more quickly, enabling your team to address them and thus further minimizing risk.
The second factor is also quite important: are there end-users available to provide feedback and are they easily accessible? One thing to keep in mind is that people are not used to the fact they will have to be available for a project throughout its duration. In the past they participate in workshops in the beginning of a project and in acceptance testing testing near the end (oh sweet memories :))
Finally you should consider how mature your team is. This means how experienced are they with agile practices such as Test Driven Development, Continuous integration, Test Automation, Collective Code Ownership (instead of finding out who broke the build, investigate and fix what broke the build), being cross technical (no specialist like the database guy, the UI guy, etc.). This however does not mean there is no room for specialists and that everybody has to be a generalist but everyone chips in where needed. A big obstacle are teams that divide work parallel with the layers instead of what i call "slicing the cake", this means implementing a feature from front to back, this will enable them to incremently deliver software.

For more information see: Agile Estimation and Planning from Mike Cohn

EF CTP5 And Unit Testing

Using Entity Framework Code First CTP5 its quite easy to write and unit test your repositories. However test should still be written against the mappings of the actual database. It would be best to split the tests that write to the database from the tests that read into a different class

Just Create a class that inherits from DropCreateDatabaseAlways<T> override the seed method to create insert the values you want.

   1: public class DBContextInitializer : DropCreateDatabaseAlways<EntityContext>

   2: {

   3:     protected override void Seed(EntityContext context)

   4:     {

   5:         ProductDummies.GetAllProducts().ForEach(p => context.Set<Product>().Add(p));

   6:         context.Commit();

   7:     }

   8: }

And then put in your Unit Test Assembly the following method:

   1: [ClassInitialize]

   2: public static void ClassInitialize(TestContext context) {

   3:     DbDatabase.SetInitializer(new DBContextInitializer());

   4: }

If you then use an in memory database it will be fast. The best way to always start from a known state (and to avoid mstest concurrency problems is to put the following code in Your TestInitialize method:

   1: [TestInitialize]

   2: public void TestInitialize()

   3: {

   4:     string key = Guid.NewGuid().ToString();

   5:     TestContext.Properties.Add(DB_NAME_KEY, key);

   6:     repository = new Repository<Product>(key);

   7: }

To delete put the following snippet in your TestCleanUp

   1: [TestCleanup]

   2: public void TestCleanUp()

   3: {

   4:     repository.Dispose();

   5:     DbDatabase.Delete(TestContext.Properties[DB_NAME_KEY].ToString);

   6: }

Unit Testing ROI

In my experience everybody knows for sure unit testing is a good thing. But then again it puzzles me why nobody is actually doing it... I think one of the biggest misconceptions is that everyone thinks that unit testing pays back over time or in the long run. This is not true! There is a more immediate payback on your investment that pays off from the very first line of code! The returns on Unit Testing grow exponentially over time...

All projects go relatively smooth the first few months until they reach a certain point where the skeletons start falling out of the closet... Regression bugs start to appear at an alarming rate, it takes longer to detect errors, more time is spent in the debugger than actually writing code, it gets harder to add new functionality,etc,...

Unit testing will:

• Reduce the time spent in the debugger
• Reduce time to test your code as a developer
• Enables you to refactor

Reduce the time spent in the debugger

if you consistently write (good) unit tests you will be focusing on smaller portions of your code and thus preventing you to have to step through your whole application. When running all your unit tests you will automatically see when you break existing functionality, like this you will hunt down the likely cause a lot quicker (probably it was that last bit of code you changed).

Reduce time to test your code as a developer

When you unit test your code those little forms with one "test" button will be of the past. An other thing which will save you tremendous amounts of time is that it is no longer needed to step through a series of screens before you actually get to the functionality you want to test.

Enabling refactoring

If you have unit tests you can refactor those methods and classes that grow an keep growing... Everybody knows these little black holes in any application that are a pain to modify and every team member is reluctant to make modifications out of fear to break something else.

NDepend Enables you to quickly identify where problems will start arising. Find below some useful queries.

Types that are to big and need to be slimmed down

WARN IF Count > 0 IN SELECT TYPES WHERE
   NbLinesOfCode > 500 OR
   NbILInstructions > 3000
   ORDER BY NbLinesOfCode DESC

for more information: http://www.ndepend.com/Metrics.aspx#NbLinesOfCode

Methods that are to big or to complex

WARN IF Count > 0 IN SELECT METHODS WHERE
  ILCyclomaticComplexity > 40 AND
  ILNestingDepth > 4

OR NbLinesOfCode > 50 //Can even be smaller
  ORDER BY ILCyclomaticComplexity DESC

for more information: http://www.ndepend.com/Metrics.aspx#ILCC

C# 4.0 Code Contracts:

I was trying out the code contracts in C# 4.0 and came across something quite surprising every time I ran my unit tests I got an annoying message box popping up “Assert Failure”, Description must use the rewriter when using Contract.Requires<TException>.

After some googling around I found that in Project Properties you need to find the code contracts tab and set “Perform Runtime Contract Checking” to full.

I opened my project properties I realized I did not have the Code Contracts tab… So in order to enable this tab you will need to install Code Contracts from DevLabs (here: http://msdn.microsoft.com/en-us/devlabs/dd491992.aspx).

Rebuild and he complained about the Assembly Mode Setting not being appropiate. After Changing it I could run my tests without having to click every time a testmethod came in the method that had the contract.

Problem solved!

Interface Contracts

One problem I came across is that when trying Interface contracts I had an assembly that ended in Contracts and apparently it does not work (Code Contracts will compile its contracts in a assembly named <assembly name>.Contracts.dll. I renamed the assembly to IContracts and everything worked as expected

Lean Principle: Eliminate Waste (Muda)

In The Toyota Production System Taiichi Ohno says that capacity = work + waste. And that one should strive for absolute elimination of waste. When  you are thinking about the absolute elimination of waste one should always keep the following 2 points in mind:
1. Improving efficiency makes sense only when it is tied to cost reduction. To achieve this, we have to start producing only the things we need using minimum manpower.
2. Look at the efficiency of each operator and of each line. Then look at the operators as a group, and then at the efficiency of the entire plant. Efficiency must be improved at each step and, at the same time, for the plant as a whole.

The efficiency can be quantified in the following fashion: efficiency =  work / cycle time.

At Toyota they identified the 7 following types of waste:

1. Overproduction
2. Unnecessary transportation
3. Inventory
4. Motion
5. Defects
6. Over-processing
7. Waiting

Now in order to eliminate waste we first need to identify waste. It is my experience that there are some recurring problems when in comes to developing software. When looking for waste we need to look at it at 3 different levels. In software developed we can look at it from a Team Member’s perspective, Team and process.

Overproduction

Overproduction happens each time you engage more resources than needed to deliver to your customer. For instance, large batch production, because of long change over time, exceeds the strict quantity ordered by the customer. For productivity improvement, operators are not required to produce more than the customer needs. Extra parts will be stored and not sold. Overproduction is the worst muda because it hides or generates all others, especially inventory.^{[citation needed]} Overproduction increases the amount of space needed for storing raw material as well as finished goods. It also requires a preservation system.

In software production overproduction can be seen as developing features that are never going to be used. At XP2002 Jim Johnson of the Standish Group reported that a study on the functionality in software. The graph below clearly shows that 64% of the functionalities build is barely used! In the past I had a lot of discussions with my colleagues about this… If a customer asks something its not necessarily mean that he actually needs it.

“If I had asked people what they wanted, they would have said faster horses”

Henry Ford

One way to avoid this is to release as early and as often as possible. This enables immediate feedback from the actual end users. Overproduction is quite frequent in when the QA department is not a part of the team. Since in most companies the QA department usually has more work than it can cope with people try to produce large batches to send to them only adding to their workload hiding possible defects. In the past I worked closely with a development team to enable them to release multiple versions a day to the test environment. This immensely improved the quality.

Something that pays of (with interest) is to invest in your deployment strategy and not wait until the very last moment. I have seen it to many times that this is neglected and every time a new environment needs to be deployed it takes a lot of manual manipulations of various configuration settings making deployment a very erroneous activity. To counter this people would only release large batches of bug fixes and features. This obfuscates a lot of problems like a bug that works on a developers machine but not on an other environment. In the worst case these problems only show up when deploying to the production environment  -since in most companies this is done by different people who are not aware of configuration changes for example – causing downtime.

Unnecessary transportation

Each time a product is moved it stands the risk of being damaged, lost, delayed, etc. as well as being a cost for no added value. Transportation does not make any transformation to the product that the consumer is supposed to pay for.

In software development the product is not only the piece of software we have written but more so the knowledge required to operate and write the software.

According to Poppendieck transportation are handoffs of knowledge . When transferring knowledge from one person to another the most effective way of doing this is by face to face communication. far less effective – in declining order – are phone calls, emails and documentation. Examples of this are requirements documents, bug reports. This however does not mean it is a waste to write them, it means you have to keep them as lean as possible and facilitate communication between the people that have the knowledge.

Even with face to face communication a lot of knowledge is lost. To quote Freud:  “The things that go without saying are usually left unsaid…”. This can easily be perceived with playing a game somebody says a sentence to the person left of him who in his turn tells it to the person to his left and so on. Could be done in the form of “I am going on holiday and I take the following things with me:” and everyone adds one item to the sentence and the next person has to repeat the whole list, if he makes a mistake he can no longer participate in the game. See how long the list is before every one made a mistake, than compare it to your requirements documents.

Other sources of waste can be branching (if its to complicated) since it also requires transportation. If you have to many branches the knowledge is lost every time you have to transport it from one branch to another. Integrate your code early and often! It is a lot better to merge your changes on a daily basis than on a 2 weekly basis!

Inventory

Inventory, be it in the form of raw materials, work-in-progress (WIP), or finished goods, represents a capital outlay that has not yet produced an income either by the producer or for the consumer. Any of these three items not being actively processed to add value is waste.

Take for example  a developer sends of some code to the deployment team who has the responsibility to deploy to the test environment than from the developer’s perspective the work is done. Then its somewhere in the queue of the deployment team waiting to get in the queue of the test team, building up inventory. Then when the testing team finally got round to testing it they discover some bugs and the functionality needs to be reworked. This largely depends on the teams definition of done. If an impaired Definition is used a team will never reach an hyper productive state. This is usually reflected in Acceptance testing cycles only just before each release (encompassing several iterations).

To avoid inventory to build up remove impediments and allot some time to “Groom” the backlog (e.g. Remove items from the backlog that are older than 16 weeks). This is a very hard concept for people to understand but don’t build up inventories. If a backlog is consistently prioritized the items older than a certain period nobody will probably remember them after one year anyhow.

Motion

As compared to Transportation, Motion refers to the producer, worker or equipment. This has significance to damage, wear and safety. It also includes the fixed assets and expenses incurred in the production.

This can be taken quite literally. One of the biggest tips I can give you is sit the Team together! This facilitates osmotic communication, the added value of removing physical distance is immense even with distances more than 2 meters. If somebody has to turn or get up from his desk to ask a team member a question he will not do it.

Osmotic communication means that information flows into the background hearing of members of the team, so that they pick up relevant information as though by osmosis. This is normally accomplished by seating them in the same room. Then, when one person asks a question, others in the room can either tune in or tune out, contributing to the discussion or continuing with their work

Cockburn 2004

A lot of teams underestimate the impact of context switching. The productivity of someone declines rapidly if he or she are combining multiple tasks. As the graph shows below multi tasking kills productivity. When someone is working on a single project his productivity is around 70% the remaining 30% is absorbed by something Mike Cohn good citizenship (socializing with colleagues, administrative duties, reading mails, …). This is in line with what Henrik Kniberg calls “Focus Factor”.

Source: Steven C.Wheelwright and Kim B.Clark (1993), Revolutionizing Product Development: Quantum Leaps in Speed, Efficiency and Quality

Defects

Whenever defects occur, extra costs are incurred reworking the part, rescheduling production, etc.

When defects appear these need to be  detected and resolved as quickly as possible.  Find below a table of a the relative cost associated with the stage a defect is detected:

Requirements development (Vision)	1x
Design (Writing the User Story)	2-3x
Construction (Development)	5-10x
System Or Acceptance Testing	8-20x
Production	68-110x
	Boehm 1981; Grady 1999

Usually the primary goal of testing is to discover errors, but teams should strive to do testing to prevent errors. This is what XP, TDD and BDD propagate, all code flows from a test. Once you are a TDD adept like me there is no other way you want to write code. It is usually very hard to convince people about the added value of writing test code (they see it as wasted effort). Invest in automated test frameworks and/or tools.

A great tool to signal an abnormality is some kind of “Andon” a visual indication that something is wrong visible to the whole team. If a build fails the first thing anyone who notices this should Stop what he is doing Fix the issue and investigate the root cause of the failure. And not look who checked in the last bit of code and points his finger in the direction of whoever caused the failure. Continuous Integration asks for a fair bit of discipline but is worth its weight in gold.

The purpose of continuous integration is not have a 100% success rate, studies have shown that the sweet spot is around 67% success rate to maximize productivity (according to Jeff Sutherland). Examples of Andon’s are for example CC tray, Team Build Notification, small tools running on an unused piece of hardware that is visible to everyone, lava lamps, traffic lights,… less obvious examples are Scrum/Kanban Boards.

If an abnormality is detected it should be fixed immediately otherwise the broken windows syndrome will kick in. This means that the defective state will be considered as a normal state.

Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it's unoccupied, perhaps become squatters or light fires inside.

Or consider a sidewalk. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of trash from take-out restaurants there or breaking into cars.

Broken windows: James Q. Wilson and George L. Kelling (1982)

Over-processing

Over-processing occurs any time more work is done on a piece than what is required by the customer. This also includes using tools that are more precise, complex, or expensive than absolutely required.

Spending more time on something does not necessarily augment the quality nor does it add value. Do just enough. Developers have a tendency to over engineer solutions – also known as gold plating. In my experience most often too much time is often spend on premature performance optimizations, trying to solve concurrency resulting in complex multithreading code that quite quickly become complex and buggy choking points in an application. I frequently encounter developers that suffer from the not invented here syndrome, writing their own data access logic, logging, user controls… These things have been done over and over again by people before them but they stubbornly believe they will do it better than the legions of developers before them.

Writing code that can become useful in “future”. One thing they forget is that if you make this consistent behavior their might not be a future for your project. I always try to keep de following sentence in mind “Simple is already complicated enough”, or to quote Einstein “Make things as simple as possible, but not simpler…”.

Successfully delivering software projects are not achieved by writing code that is really cool to write but by just restricting yourself to the bare minimum to turn a requirement into a working bit of software. In the passed I was writing a kind of address book application that had to synchronize data from the ERP system to active directory. After struggling a few hours trying to write a windows service that had to run on regular intervals, I wound up writing a console application that I scheduled in the windows task scheduler.

Another thing that falls under this category is writing unnecessary code. Whether you have been writing this code or the code is generated by a tool (Code generators, Snippets, T4 templates etc, etc,…) is all the same. If you have repetitive patterns in your code try to look at design patterns, object oriented principles whatever and refactor them out. Every line of code that gets into your code base is a line of code you drag along for the rest of the life of the product.

Waiting

Whenever goods are not in transport or being processed, they are waiting. In traditional processes, a large part of an individual product's life is spent waiting to be worked

Waiting is an obvious waste, although an obvious one a very common source of waste and they reside very stubbornly in an organization or process. I can not count the times I heard there is nothing we can do about that, it always was this way. Although upon prying to the root cause no good reasons can be given to sustain these impediments. Often very strong indicators of suboptimal communication path ways by sending mails, filling out forms… Waiting occurs a lot in teams that do not successfully self organize or are not cross functional. Bottlenecks are often enforced by formal approval procedures, the lack of mandate by a team, lack of access to the QA department that has to perform Acceptance testing. These impediments can however be easily overcome by a simple value stream map which will quickly show bottlenecks in the process.

References

• Agile estimation And planning by Mike Cohn
• Lean Software Development: From Concept To Cash by Mary and Tom Poppendieck
• The Toyota Production System by Taiichi Ohno
• Wikipedia

Lean Principle “See the whole”

The remaining work report showed a clear buildup in items that were “Ready to Test” indicating a bottleneck somewhere in the process. I was wondering what was going on… I was under the assumption that the tester did not have any time to test these items (he had some work with existing customers who had to generate some kind of annual reports, so I was waiting for this surge to end). But after asking the tester what was going on he was quite surprised and responded that he only had 12 items that were ready to test instead of the more than 40 shown in the report. After some asking around the root cause of the problem was that the development team did not release a version to the test environment yet so the tester was not able to test the other items.

This example clearly shows that the reporting help you in making bottlenecks visible. However they don’t show you exactly what the root cause is. You always have to keep in mind to See the Whole.

Product Backlog Pareto

Pareto analysis on Product Backlog Items and Bugs

This report helps you analyze the “health” of your backlog for more information:Implementing Lean Software Development: From Concept to Cash (page 170).