TestNG NG, Next Generation of a runner for TestNG

I’ve been working with TestNG for 2 years now. The team that I work on made a decision to switch to TestNG thanks to one very important feature, @BeforeSuite and @BeforeClass. We are using Gradle as our build tool. Gradle supports TestNG tests execution.

With all the great features that TestNG comes with it also comes with features that could obfuscate test code readability. We also observed that there is no guarantee that the tests in the same class file will be executed together as a set.

With 25 minutes time of my train ride to/from work and determination to build something sweet and simple, I decided to create a Gradle plugin that will run TestNG tests in deterministic order, supporting only a small set of TestNG features.

I’ve made the code available here: https://bitbucket.org/gigu/testngng. The code is still work in progress, however there is already a functional Gradle plugin that can produce same style of XML reports as TestNG itself. There is also option of a Html report with simple and pretty style. I’ve written it in Groovy as I like Groovy. It’s Open and available to anyone.

Features supported by TestNGNG

  1. TestNGNG will recursively scan class folder passed in as parameter, in search of possible tests files. It treats this top-level folder a Suite.
  2. TestNGNG ENSURES all the tests in a class file are executed as a set of tests.
  3. TestNGNG will build a tree of tests and dependencies at the beginning of a run, before it executes a single test. The test tree is build in a form of a: One Test Suite -> Many Test Classes -> Many Tests.
  4. TestNGNG supports original TestNG annotations, it doesn’t have it’s own annotations as it is only a runner.
  5. TestNGNG support @Test annotation of a method or a class.
  6. It supports dependencies between test methods with:
  7.  It supports @BeforeSuite/Class/Test/Method and @AfterSuite/Class/Test/Method. However, @BeforeTest, @AfterTest and @BeforeMethod, @AfterMethod are treated in the same way and executed before/after each test. Just to avoid (or add to) confusion.
  8. It supports disabling of a test by enabled attribute of an annotation:

    I’m not proud of this feature though and am tempted to remove it.

  9. It supports exception expectation by expectedException attribute:
  10. TestNGNG supports data providers, however they have to be declared within the same Test Class file. Data providers could be named, or anonymous. For example:
    @Test(dataProvider = "makeMeSomeData")
      public void testWithProvider(String v1, String v2){
      System.out.println(String.format("%s - %s ", v1, v2));
    public Object[][] makeMeSomeData() {
      return new Object[][]{
        {"some1", "Some2"},
        {"some3", "some4"}
  11. TestNGNG supports setup methods inherited from base classes. For example:
    public abstract class BaseForTestWithTestSetupMethods {
        public String baseValue = "";
        public void executeBeforeSuite() {
            baseValue = "BeforeMethod";
    public class TestClassExtendingFromBaseClass extends BaseForTestWithTestSetupMethods {
        public void shouldPass() {
            assertThat(baseValue, is("BeforeMethod"));

Gradle plugin

Gradle plugin is very simple to use. It only requires plugin jar on a class path and TestNG as a testCompile time dependency. The plugin adds testngng task to the project. Sample use of a plugin:

buildscript {
    repositories {
    dependencies {
        classpath fileTree(dir: '../gradle-plugin/build/libs/', include: '*.jar')
        classpath fileTree(dir: '../testngng/build/libs/', include: '*.jar')
        classpath group: 'org.codehaus.groovy', name: 'groovy-all', version: '1.8.0'
apply plugin: 'java'
apply plugin: 'testngng'
repositories {
dependencies {

Most of the other TestNG features are not covered as they are useless and very often stand in a way of tests readability and simplicity.

Don’t ask what you test framework can do for you, ask what can you do for your test framework.

How to build the plugin and use it

You would need to checkout the project from a BitBucket and build it with Gradle by typing from command line:

gradle jar

The build jar will be used by test runner Gradle plugin inside inside the testngng_gradle_sample project. You can ran gradle command from the sample project folder:

gradle testngng

I would love to see someone having a go and providing me with some feedback. There is still a whole lot of stuff on my list that needs development in the near future, like a build tests running feedback, more plugin configurations and multiple suites per project/module.

Cheers, Greg

Tests revealing design problems

Within last few months I was introducing tests into untested code. It was not old code, just some code that was not Unit Tested. As much as I was eager to do so, I found myself getting in a bit of a pickle.


Code complexity

Code was actually quite complex and difficult (smell) to understand. I was also a little scared that I will misunderstand the code and its intention. I assume that I will not be the only person facing this problem while working on the piece of code.

Fortunately there was some testing coverage in a form of a higher level Acceptance tests (not sufficient though).

I decided to refactor the code a little to make it more manageable. Extract few methods, rename few variables, remove unnecessary conditions and loops, etc.

When I finished refactoring and made sure build was still successful, I checked in the code and off I go to write some Unit Test. That was the point when I hit another brick wall.

Instantiated dependencies

Necessary dependencies in a class that I wanted to test were not provided to an object upon creation, but were instantiated internally. This means that code is not flexible to changes in a future and is impossible to test (smell). Making all the dependencies passed through constructor will make it more obvious to what the class will do, and make it possible to unit test.

So, after some refactoring again, building and submitting the code, I had my dependencies all wired up through constructor, just the way I like (mmmm, delicious).

Concrete classes

Many of the dependencies were still concrete classes. Pity, actually you don’t care what’s in the guts of your collaborators (most of the time) as long as they do what they indicate. This could be replaced by interfaces (or some abstract types in worst case) as dependencies. They also make it easy and simple to stub or mock your collaborators, so the testing is simpler to setup and you’ll be testing only the correct object.

Another refactoring session ended in a bunch of new interfaces.

So I started to write a test. I created my object and started to pass some dependencies. The problem was that I had to pass some other dependencies to those dependencies, and dependencies of dependencies and even more dependencies and … you know where this is going.

This great monster had too much setup code than the one I was going to test.

Too many collaborators

This got me into thinking that something was not right. The thing was that the tested class was trying to do too much. This was the main reason of mentioned previously code complexity. Testing was not easy and possibility of introducing new bugs during future changes was great.

So, extracting few methods here and there (around same functional area) and I was able to pull those methods into new object. This new class was very simple and was responsible for one area of functionality only. I could unit test that new class.

Last problem during the tests was that I got some unexpected behaviour. As I looked into the code it turned out that there was one more smelly thing left.

Static methods

There was some code that was calling to a static method of another class. It was impossible to mock or stub it. What’s even more horrible, it was using some internal static variables. That caused some un-expected behaviour.


As it turned out, I found problems in a code even before I started to write a single line of test. Just thinking about the ways to test, revealed first issues. More of the issues appeared while testing. The work I had to do in order to make code tidy, simple and testable could be avoided by writing it in TDD/BDD way.

I’m convinced there could be more problems revealed. Here are these that I found once again:

  • complex code
  • instantiated (new Foo()) dependencies inside tested class
  • concrete classes with no interfaces
  • too many collaborators (class was doing too much)
  • static methods (brrrr)