When done is DONE (or not)

Not too long ago I had a conversation with one of the senior member of the management team of project I’m working on. I had some ideas on how can we do things faster by improving our testing (not developer testing but end-to-end, QA and regression testing). After few minutes of conversation I was asked a very basic question: “What is the definition of DONE on our project?” I was just going to open my mouth and jump out with an answer like: “Well, it takes us usually 3 days to develop piece of functionality”, but I stopped. I actually wasn’t sure. We spend few more minutes discussing some other issues but when I left I felt that this question is still on the back of my mind, trying to desperately find the answer.

After number of attempts I decided to rephrase the question. What is the Goal of the project? Couldn’t find a simple answer. So I asked even more general question. What is the goal of the company? I recalled books by Eliyahu M. Goldratt, “The Goal” and “The Race”. “The goal of the company is to make money in the present as well as in the future”. The goal is to win the race for customers.

My team works on creating and maintaining the software that produces the data for other systems within the company. Those systems are used to deal with clients, to provide them with reports, to sell them information, to protect client interests. This means that our project indirectly contributes towards company’s goal.

All the other teams and projects that are receiving data from us are our customers. We should make all the effort to deliver the necessary features to the consumers in a timely manner as they will be use to generate the revenue.

That’s it. This is my understanding of DONE.

In other words:

  • It’s NOT DONE when BA finalizes the requirements and forms them as stories that got accepted by all stakeholders
  • It’s NOT DONE when developer finishes coding the solution and fixes all the bugs
  • It’s NOT DONE when QAs, BAs finish testing and approve the deliverables
  • It’s NOT DONE when Downstream systems receive the data and confirm it’s quality
  • It’s DONE when client receives the service that he or she requested thanks to the piece of software my team delivered. That’s when it’s DONE!

I think there is an important aspect to touch on. It is the effort of the entire team before DONE could be announced. No one should silo himself into a specific role and take responsibility for the specified area only. Developers should help with delivery of tools for release and testing automation, BAs should help with testing, QAs should help to form requirements, etc.

So, next time when you think you’re DONE, think again. Perhaps you are not really there yet but you could help someone else to make it happen.

Wish you all many happy DONEs in the future. Greg

Gradle IDEA plugin, configuration and usage

We are using Gradle as a build and release tool in our project. It’s flexible and does exactly what you expect from it, with no xml configuration magic.
Gradle IDEA plugin gave us a way of generating IntelliJ project and modules files out of the freshly checkout source with a single instruction.

Gradle IDEA plugin

All we needed to do is to include the plugin in out multi project setup and run the gradle idea tasks from the command line.

allprojects {
…
  apply plugin: ‘idea’
…
}

First time when we generated the files they were valid, however the module configurations needed few simple fixes. The IDEA plugins provide hooks into project and module generation that we used to amend the generated IntelliJ files.

Setting the Java version on the project


allprojects {
…
  ideaProject {
    javaVersion = '1.6'
  }
…
}

This code snipped sets version of JDK on the whole project, which in turns makes it ready for compilation in IntelliJ.

Moving module dependencies to the top of the list

Gradle IDEA plugin creates dependencies for all the modules, however it moves the dependencies between modules to the bottom of the list. For example, if module B has dependencies on module A plus some of it’s own JAR dependencies, the JAR will appear before the module A dependency.  It was problematic for our project.  The snippet below is using module configuration hook to reorder module dependencies to the top of the list.

allprojects {
…
  ideaModule {
    whenConfigured { module ->
      def moduleDependencies = []
        module.dependencies.each {
          if (it.class.simpleName == 'ModuleDependency') {
            if (it.scope.equalsIgnoreCase("COMPILE")) {
              moduleDependencies += it
            }
          }
        }
      module.dependencies.removeAll(moduleDependencies)
      def jarDependencies = new LinkedHashSet(module.dependencies)
      module.dependencies.clear()
      module.dependencies.addAll(moduleDependencies)
      module.dependencies.addAll(jarDependencies)
    }
  }
…
}

Adding defaults to Run configurations

We have also discovered that it was useful to add some extra defaults to Run configurations and to TestNG running configurations.

We did this with Idea Workspace hook and a bit of XML injection. Code bellow is the sample snipped we used.


allprojects{
...
  ideaWorkspace {
    withXml { provider ->
      def applicationDefaults = provider.node.component.find { it.@name == 'RunManager'}.configuration.find { it.@type == 'Application'}
      if (applicationDefaults != null) {
        applicationDefaults.option.find { it.@name == 'VM_PARAMETERS'}.@value = '-Xmx1024m -XX:MaxPermSize=256m -Xss64k'
        applicationDefaults.option.find { it.@name == 'WORKING_DIRECTORY'}.@value = 'file://$MODULE_DIR$'
      }
      def testNGDefaults = provider.node.component.find { it.@name == 'RunManager'}.configuration.find { it.@type == 'TestNG'}
      if (testNGDefaults != null) {
        testNGDefaults.option.find { it.@name == 'VM_PARAMETERS'}.@value = '-XX:MaxPermSize=128M -Xmx1024M'
        testNGDefaults.option.find { it.@name == 'WORKING_DIRECTORY'}.@value = 'file://$MODULE_DIR$'
      } else {
        def clonedNode = new XmlParser().parseText( XmlUtil.serialize(applicationDefaults))
        clonedNode.@type = 'TestNG'
        clonedNode.@factoryName = 'TestNG'
        provider.node.component.find { it.@name == 'RunManager'}.append(clonedNode)
      }
    }
  }
...

}

This snippet sets some default JVM parameters and working directory for running configurations.

The snippets made it possible for anyone to quickly get up and running on any clean machine with checked out project sources.

Cheers, Greg

Build script and project level variables (Gradle)

Following on my Introduction to Gradle here is a simple tip on script level variables.

Variables

I found out today that declaring something on the top level of a project build file doesn’t mean that all the Gradle blocks will see it. The problematic one is buildscript {}. It happens as buildscript section is evaluated before any other part of the script is.

So, for example:

myVariable = “I was here”

allProject {
  println “It will be here $myVariable”
}

buildscript {
  println “But it won’t be here $myVariable”
}

Defining variables on top level could be useful as they could be treated as constants and reused throught the script (I use those for repository URLs for example)

The easiest and simplest solution would be to put buildscript{} as a first block and declare all the variables inside.

buildscript {
  myVariable = “I was here”
  println “Here $myVariable”
}

allProject {
  println “.. and here $myVariable”
}

Buildscript block doesn’t have to be at the top but I feel it’s more elegant and clear as all the constants are on the top of the file that way.

Introduction to Gradle

For last few weeks I’ve been moving rather large collection of Ant, Shell and other scripts into Gradle. In my opinion Gradle is great. It has two things that I value the most when it comes to tools:

  1. The build scripts are CODE. As scripts are CODE I can do everything I could do when I code. It gives me freedom from XML
  2. It has dependencies management powered by Ivy.

Scripts are Code

The build scripts are Groovy code. Groovy is dynamic language that runs on JVM (similar to JRuby in some sense).  This means that I can test my build scripts with unit tests. This also means that I can do loops and conditions easier without weird, artificial XML tags.

Dependency management by Ivy

Gradle team, instead of inventing new dependency management, actually decided to use Ivy. Simple and easy tool that does what it says on a tin, nothing more. More important it will not try to download the Internet every time it runs.

The beginnings

First of all, if you would like to follow me with this article, grab yourself installation of Gradle. It is self-contained zip, and the only thing that you would need is Java installed.  Verify your Gradle installation by typing gradle –v from your command line.

Gradle expects your build scripts to be in a project folder in a file called build.gradle. You can write your scripts in another file but every time you execute Gradle you will have to specify where your build script is: gradle –b file_name.gradle.

To execute a task defined in build.gradle file you need to type: gradle taskName.

To see the list of all available tasks in for project type gradle tasks from command line.

Write you some tasks

Gradle has a concept of a task. Task is combined of multiple actions. One can define a task as new and assign some actions to it, or use predefined Types of task (Copy, Exec, etc.).

Example of new task definition:

task doStuff << {
  // Hey hey, I’m doing stuff
  println ‘Doing stuff’
}

Example of task that copies files:

task copySomeThings(type: Copy) {
  from ‘source’
  into ‘destination’
}

There is a very subtle difference between those two ways of defining tasks, and I got caught many times by it. The first task, doStuff doesn’t have predefined functionality. Calling method task doStuff creates task with name doStuff. However, I need to define some actions. This is the role of operator << (shift left). What actually happens is:

  1. Task doStuff is created
  2. Action that is defined in closure (between two curlies, {}) is appended to the end of the actions list of this task.  First and only action that will happen will be printed out into STDOUT.

If I need another action to follow the thirst one I defined I could do:

doStuff << {
  // Wow, now I’m doing another action
  println ‘Weheeeey’
}

The definition of copy task however doesn’t contain << assignment. It does contain the closure though. This closure is passed into the task Copy to configure it. from and into are methods defined on task type Copy that can be used to specify where to pick up files from and where to drop them. Every predefined task has some parameters and methods that I can specify. Please refer to documentation  (link to Gradle task list) for details of task.

This is important

The most common mistake I kept on making was defining a custom task and forgetting the << operator. The task did nothing and I was spending countless hours investigating why.

task doStuff(){
  // this will not happen
}

Task dependencies

In Gradle I can define dependencies between tasks. This means that when I want to execute specific task and there is another one that needs to be executed before it, Gradle will execute it too. The way to define task dependencies:

task doStuff() << {
  // I’m doing stuff
}

task doingEvenMore(<strong>dependsOn: doStuff</strong>) << {
  // Doing more
}

You can also specify name of the task instead of the task reference. This allows for kind of a lazy evaluation of build script.

task doingEventMore(dependsOn: ‘doStuff’) << { }

If you need to add dependency for already defined task you can use dependsOn method of a task.

doingEvenMore.dependsOn someOtherTask

Or dependsOn property of a task:

doingEvenMore.dependsOn = [doStuff, someOtherTask]

There is a catch with the last method: it overrides all previously created dependencies.

Then, there was Java

As of now I was describing how to create tasks and declare dependencies between them. To actually use it on a project we will need some more action. That’s why it is time to introduce Gradle Java plugin.

apply plugin:’java’

Java plugin adds some task to your build. You can check them out by executing gradle tasks from command line. Tasks that can compile, test and even jar your project.

What we are interested in are dependencies that project might need and the way Gradle handles them. By dependencies I mean third party libraries like: log4j, junit, etc. You can see dependencies in your project by executing gradle dependencies from command line.

Dependencies

Java plugin adds few dependency configurations. I’m interested in compile time dependencies, as it is needed for my project to compile and run.

apply plugin:’java”

repositories {
  mavenCentral()
}

dependencies {
  compile group: ‘log4j’, name:’log4j’, version: ‘1.2.9’
}

Inside repositories block I used method mavenCentral(). This is method predefined by Java plugin providing access to Maven central repository. As I mentioned earlier Gradle uses Ivy to resolve dependencies. Ivy in turn works rather well with Maven repositories.

Lines above will add single log4j dependency to mine project. The cool thing about it is that it will also resolve all its dependencies.

We can also add multiple dependencies by doing:

dependencies{
  compile(
    [group:’log4j’, name:’log4j’, version:’1.2.9’],
    [group:’xfire’, name:’xfire’, version:’1.2.6]
  )
}

This will add two dependencies to my project.

There are few more dependencies (runtime, compile, testCompile, testRuntime, archive) added by Java plugin. Dependencies are nothing more than a list of libraries (files) included in a build lifecycle (compile, test etc).

Configurations

One can define its own dependency configuration by creating new entry in configurations block of a build script.

configurations {
  gregWasHere
}

dependencies{
  gregWasHere group: ‘log4j’, name:’log4j’, version: ‘1.2.9’
}

Configurations are nice and simple way of adding compiletime or runtime libraries to any build. For example I used it when executing external Java process that needs extra libraries in classpath.

task executeSomeJava(type: JavaExec){
  main = “com.some.main.Bla”
  args = [“1”, “2”]
  classpath = configurations.gregWasHere
}

Summary

Gradle is really great and if you are still stuck in the world of XML and hate it, give it a try. Few tips I written in this article, could be used as a good starting point. I also found that coding to task convention that Gradle uses, simplifies the build and release process.

Few good resources to read when starting with gradle:

Broken Maven repository and Gradle dependencies

Few days ago I got into trouble when building project using Gradle.

Created deployables (fat jars) looked fine when I build it on my local machine, however when I build on remote box (Linux) I was missing some dependencies. The build scripts were exactly the same. The only difference was: the remote box didn’t have local Maven repository. This shouldn’t cause any problems as remote repository should be the same.

Gradle, a better way to build

However 🙂 on the project that we are working, the remote repository is managed by team and it is in our internal network. It also doesn’t proxy to Maven repository.

Some of the artifacts that got uploaded to that repository were missing pom files. When Gradle (via Ivy) was resolving the dependencies it created pom file on the fly with no dependencies on other artifacts.

When I discovered the problem I uploaded missing pom files. It didn’t fix the problem though. As it turns out, Ivy cached the self created pom files in USER_HOME/.gradle/cache folder. Running gradle with -C rebuild option doesn’t solve the problem as this option only clears the cached Gradle scripts.

When I deleted the folder USER_HOME/.gradle/cache, Gradle got all the dependencies resolved properly!

I was using Gradle milestone 3.

I hope you won’t get stuck on the similar problem as I did. Maintain you repository or use public.

Greg

Design release process from very beginning

Creating good software that meets customer expectations is 50% of success. The other 50% is getting it to the customer in a most quick, easy and smooth matter as possible. This process is called releasing software.

breaking chain

Release flavours

Releasing software comes in two flavors, one is a clean slate installation, second is version/maintenance upgrade. Where first one is rather simple the latter could be quite problematic. The more complex the application and environment it runs in, the more problematic it’ll be.  Releasing software to running cluster of application servers with data migration and without downtime could prove challenging for the toughest release gurus.

I always thought of release in a same way I receive or send parcel. When I have something to sell I organize it online, Delivery Company comes to pick it up and it’s on its way. I can then track it online to see the progress my parcel is making.

It is exactly how I would like any release process to be. When my package (new version of software) is ready I would like to press the button and have it on its way. All I should do is monitor if something went wrong and what if it did.

Automation

Automation of the process is a must if one wants to remove wasted time on same repeatable steps. It is quite often automation that gets neglected from the beginning of software life, until the point when release becomes unmanageable manual monster.

Automation should actually be planned from the very beginning. Whenever I start to work on a project I ask myself what do I need to do to get the software to end user as quick as it’s ready with as less possible effort. This is the beginning of the release automation adventure.

Releasing often and quickly adds into the whole iterative development process, combining the whole as one iterative delivery. Process reduces time that takes for the requirement to reach end user as a piece of functionality. That makes end user happy and gives way to feedback. Software development team can use feedback to adapt and evolve the software.

So, what now?

It is important to design and implement release process from the beginning. It is also important to automate it. It will make it possible to ship as soon as piece of work is done. I know of teams where releasing is happening on a daily basis.

Automate every step that requires human interaction. Look for candidates in places where the only manual thing is pressing a button or running a script.

If release process requires monitoring (e.g. waiting for application to stop before continuing release process), try to automate that as well.

I also found that usage of tools that support coding the process, not configuring it via XML, works much better and are more flexible and easy to use.

Little creative fingers

We all have heard about creativity but did we ever take a moment to think about it for a while. What it really is or what it means? How to be better at it?

Creativity

Definitions of creativity are actually rather simple. Wikipedia states that Creativity is the ability to improve, adding value. Google quotes Princeton University’s website saying that it’s “The ability to create”.

Creativity is a mental process, involving discovery of new ideas, or new view on the old ideas. The process is powered by our conscious and unconscious mind.

With today’s modern psychology and cognitive science, creativity is still rather unknown field. There are number of theories on the process, but all of them can’t explain it precisely.

Graham Wallas presented his theory about creative thinking in 1926. He is dividing creative process into following steps:

  1. Preparation – work on a problem that involves understanding it and exploring different views on it
  2. Incubation – where the problem is injected into unconscious mind
  3. Intimation – it happens when one get the feeling that idea is going to emerge soon
  4. Illumination – where the creative idea bursts into conscious mind
  5. Verification – when the idea is consciously verified, elaborated and applied

My own findings

I started with reading some book on the topic and browsing the Internet. Then I saw TED talk by creativity and innovation expert, Sir Ken Robinson. He inspired me to observe my 3 years old daughter when she was playing. They say that small persons mind is like a sponge. It soaks in all the new encounters.

Things I noticed

A lot of my daughter’s new toys emerged from an item of everyday use, used in a completely new way (laundry basket became a boat, toilet paper roll became a telescope, etc.). She has a great skill to take an item, turn it around, look at it from different angle and than decide what new toy it will be.

My daughter is very good girl and she is hardly ever destructive  (I only assume that boys are more talented in this area). She grabs things that are around her and puts them together. New creation is inspected and either accepted as new toy or discarded (sometimes put together in different way).

Kids have more simple/different way of looking at things. This is what makes them so creative. When “Little Prince” saw picture of a hat, he said it was a picture of giant snake that swallowed elephant (perhaps he was right).

Thinking about the way my daughter creates new toys I came to conclusion that creative adult thinking is very similar most of the times. We look at the things from a different angle, look at it upside down, take tools that we have never used to do different sometimes odd job (using shoe as a hammer).

Putting things together is less physical, more mental. It involves taking pieces of information, knowledge, experience, and combine them together to produce something new. A book, real life observations, notes and mind maps turned into this article.

Recall any music interview that you might hear. Musicians typically mention who is their inspiration. Normally they mention another musician/music style etc. You can recognize in their music, tunes from other musicians, same musical patterns, instruments, etc. This is being creative by combining all the bits together in a new way.

Creativity and software development

Above description of creative processes are sounding very familiar to what I’ve been doing everyday creating software. All my experience from working with other people, technical knowledge I have, domain knowledge I acquired and everything else I know influences the code that I write.

I found that knowledge of different languages and patterns that are present in those, helped me to bring new ideas, become more creative at what I was doing. C# experience helped me to introduce some new patterns to Java. Dynamic and functional languages gave different view on type safety and state. Tools that I typically used in one technology, gave new light to a better use of tools in other technology.

I also found that being brave enough to try new things, use of different tools, led me to a new discovery. For example, swapping build tool to brand new, helped defragment not fully automated release process.

Summary

Everyone can be creative however it doesn’t mean that it is simple. More information you have (including domain, technical knowledge and everyday life things) is helpful. More willing you are to try new and different things the better.

Try to stay open to information but be careful, as it is easy to get overflowed with it in the age of Internet. Be open to new things and listen to other’s ideas, don’t shut them down immediately.

Handfull of links (somehow references)

Books:

Internet:

Organisation change in disguise

 

There is a lot of tweets in the Twitterverse about changing organisations, people, processes. Not too long ago I was a consultant with “change everything” mind set. I was also surrounded by like minded people. We tried together in our crusade against bad, bad things in software development industry. I don’t feel we have achieved substantial success.

After many attempts I realised the biggest problem that we have never tackled properly. PEOPLE ARE AFRAID OF CHANGES. This rather simple psychological thing has no technical or economical background.

However, there is something I observed during my attempts at the place I work.

 

On the other hand I observed that adopting different approach in the place I work at the moment produces better results.

I started as usual with the same consulting spiel of “we need to change“. Even with the people who saw problems it was hard to convince them to change. I got a little bit demotivated at work.

Disguise

Coming back from holidays gave me new energy and new will to tackle problems. This time however I change the spiel from “we need to change” into “how about I improve this thing by doing that“. I found that in most cases there was very little resistance for improvement. Very often I tried to hide the change behind the innovation.

So, my little advise is: don’t change where/what you can’t. Disguise the change behind innovation and improvement. The change will happen by itself, the difference is: people won’t be scared of it. It also goes much better with management as they fear big bang changes the most.

Greg

Innovation, academia VS industry

Last week I was presenting the paper me and my collogues wrote. You can find it here: Enabling Testing, Design and Refactoring Practices in Remote Locations.  The paper was accepted for ICST 2011 conference workshop, RefTest. It is a short paper about problems and some solutions for practice transfer in distributed team.  However this is not what I would like to write about.

The workshops gave me something to think about. The whole conference was targeted at bridging industry and academia together. Unfortunately at the track I attended there was only two papers and presentation from the industry (one was mine).

I was hoping to hear some new innovative ideas that I would be able to take into my everyday work. After a whole day of workshop sessions I was left with no extra ideas. Plenty of smart academic people were presenting research that I found rather useless in everyday life.

I was also little disappointed after the presentations, as none were actually summarizing the research in a way useful for the industry.

I thought of all those blog articles, posts, tutorials, video presentations, etc. from industry people with the ideas and the way of using it. Of all the technical innovations, processes, practices that got discovered in last few years.

Does it mean that academia lost their leading role in software development industry? There is very little in press about new discoveries in the field of software development.

Thoughts?

Greg

 

Scalable Lean and Agile, QCon London 2011

qcon-londonToday was the first day of QCon 2011 in London that I attended. I thought about sharing what I remembered while it is still fresh in my head (and the head is buzzing with ideas).

Craig Larman spoke at opening note and at one of the sessions. Embarrassing that it is, I never heard about Craig before, neither I read any of his books. I did enjoyed the way he presented his material.

Craig was talking about scalable, distributed agile teams. Apparently he has a lot of experience working in this kind of environment, gathered throughout many years of his professional career.

Craig described a typical situation where delivery teams are physically located in different countries, working on the same project. In his presentation he provided practical tips for dealing with issues that rise in distributed environment.

The most important points that Craig delivered in his presentations are:

  • No disperse team – all the teams are sharing same goal and work as equal partners. There should be no hierarchy in teams.
  • Practices should be shared and guarded by Committees of Practice. Committee should have a leader that gets slack time to overlook the Practice and organize teams appropriately.
  • Software design should emerge as a collaborative work during workshops. Embrace open space in the office and lots of white boards.
  • Diverge and merge – teams should come out with design on their own sites, then they should hold Video, show and tell session with all the other sites.
  • When teams come with design on a specific part of the system, team leaders should work across all teams to have a common design for the whole system.
  • A documentation workshop after iterations embraces knowledge sharing and collective code ownership.
  • Avoid PowerPoint architects. Architects that are so far away from the code that they became architecture astronauts.
  • Component guardians are team members who look after code across all the teams, making sure that high quality is kept. If only one team takes care for only one component, the crap code is going to be maintained and accepted within. Teams needs to be cross component and cross-functional.
  • When components involve public interfaces, start with weakly typed, for example do ( aMap ). Once you know more and more code is stabilized you can make those more formal and strongly typed.
  • Continues integration is a must have.
  • There is a need to see the people across all teams. It builds trust and respect. Try to use cheap technology like Skype and web cams.
  • Team members should meet, rotate and travel to other sites.
  • Keep on thinking and adapting your methodologies and processes. Never retire.

Above points are mainly from the keynote Craig was giving. The second session was more about contracts and collaboration on “offshore” delivery model.

Some more interesting points:

  • Outsourcing development you will have to deal with culture, language, knowledge, domain, skills difference issues.
  • Address skills difference via internal education. Brown bag sessions or workshops made by more experienced team members.
  • Avoid single point of contact with the team. Visit teams before project kick-off. Physical contact is very important to create relationship.
  • Ask and rephrase questions in different way (or from different viewpoint) to test the answers.
  • Encourage team members to say no when needed, remove fear.
  • Don’t patronize teams capabilities by giving simple, disconnected tasks. This will reduce morale and make people leave, as they would think their job sucks.
  • Before project kick-off organize Domain Workshop (so the team can understand the domain) and Vision Workshop (so they can understand what to deliver).
  • Specification by example.
  • Make sure team is stable as much as possible (no leavers and joiners).
  • Make it clear that at the end of every sprint you are expecting a shippable product with the number of functionality developed.

Many of the things that Craig was talking about can be found in his book “Scaling Lean and Agile development”.

Many more info and a good introduction to Lean Software Development can be found at Crag’s web site. You can also enjoy a video of him telling a bad joke 🙂

Cheers, Greg