Simple Artifactory plugin for total size of Artefacts in local repositories

I’ve done a little plugin I hope might be useful for others who would like to know what is the total size of artefacts stored by Artifactory in local repository.

The functionality is quite simple:

  1. You can obtain a size of a repository by sending a POST request to url: /artifactory/api/plugins/execute/getRepositorySize?params=repoName=repository-name
  2. You can obtain a report in a form of a JSON Array for all local repositories by sending a POST request to url: /artifactory/api/plugins/execute/getAllRepositorySizes

Plugin code and a sample JSON response for all repository sizes below.

Just want to add that JFrog did a brilliant work with their very simple and elegant Artifactory API.


import org.artifactory.repo.RepoPathFactory
def getDirectorySizeRecursively(def repoPath){
def children = repositories.getChildren(repoPath)
if (children.size() == 0) return 0
def total = 0
children.each { item ->
if (item.folder) {
total += getDirectorySizeRecursively(item.repoPath)
} else {
total += item.size
}
}
total
}
executions {
/**
* This method expects parameter repoName to be passed. It will than calculate the TOTAL size of repository and return the value.
* If the repo name is invalid or not provided it will fail.
* Sample invokation:
* http://localhost:8081/artifactory/api/plugins/execute/getRepositorySize?params=repoName=ext-release-local
*/
getRepositorySize() { params ->
if (params.size() > 0 && params['repoName'] && params['repoName'].size() > 0){
def repoName = params['repoName'][0]
if (repositories.localRepositories.contains(repoName)){
message = getDirectorySizeRecursively(RepoPathFactory.create(repoName, '/'))
status = 200
} else {
message = "Repository $repoName doesn't exists"
status = 404
}
} else {
message = 'Missing repository name parameter repoName={name}'
status = 400
}
}
/**
* Prepare report on all repositories
* Sample invokation:
* http://localhost:8081/artifactory/api/plugins/execute/getAllRepositorySizes
*/
getAllRepositorySizes() { params ->
def repositoryReport = "[\n"
def totalNumberOfRepositories = repositories.localRepositories.size()
repositories.localRepositories.eachWithIndex { repoName, index ->
def repoPath = RepoPathFactory.create(repoName, '/')
def size = getDirectorySizeRecursively(repoPath)
repositoryReport+="{\"repoName\":\"$repoName\", \"sizeInBytes\": $size}"
if (index < totalNumberOfRepositories -1) {
repositoryReport+=",\n"
}
}
repositoryReport+="\n]"
message = repositoryReport
status = 200
}
}


[
{"repoName":"libs-release-local", "sizeInBytes": 122233},
{"repoName":"libs-snapshot-local", "sizeInBytes": 0},
{"repoName":"plugins-release-local", "sizeInBytes": 0},
{"repoName":"plugins-snapshot-local", "sizeInBytes": 0},
{"repoName":"ext-release-local", "sizeInBytes": 122233},
{"repoName":"ext-snapshot-local", "sizeInBytes": 0}
]

To install plugin just copy the RepositorySize.groovy file to ARTIFACTORY_HOME/etc/plugins folder.

 

Groovy Map to Json String converter

I use Groovy a lot. It’s simple and easy to use, runs on JVM and saves me from Java verbosity. I also like the fact that it’s dynamic.

I use it with a simple Servlet to return data to JavaScript AJAX requests. I like to keep the server side very simple, so I don’t create a lot of POJOs and I’m using HashMaps instead. Groovy has few convenient methods that makes it easy to work with Maps.

I do convert the maps into JSON when returning results to a browser. I don’t use anything fancy, just a couple lines of Groovy code to do that.

class JsonMap {
def toJSON(elements, depth = 0) {
def json = ""
depth.times { json += "\t" }
json += "{"
elements.each { key, value ->
json += "\"$key\":"
json += jsonValue(value, depth)
json += ", "
}
json = (elements.size() > 0) ? json.substring(0, json.length() - 2) : json
json += "}"
json
}
private def jsonValue(element, depth) {
if (element instanceof Map) {
return "\n" + toJSON(element, depth + 1)
}
if (element instanceof List) {
def list = "["
element.each { elementFromList ->
list += jsonValue(elementFromList, depth)
list += ", "
}
list = (element.size() > 0) ? list.substring(0, list.length() - 2) : list
list += "]"
return list
}
(element instanceof String) ? "\"$element\"": element?.toString()
}
}
view raw JsonMap.groovy hosted with ❤ by GitHub
class MapToJsonTests extends GroovyTestCase {
private final def mapper = new JsonMap()
void test_convertMapWithListsOfMapsIntoJSON() {
def map = ["a": "a", "b": [["b": "a"], 'd', [12, 12, "e"], ["r": 12]]]
def expected = '''{"a":"a", "b":[
\t{"b":"a"}, "d", ["12", "12", "e"],
\t{"r":"12"}]}'''
def result = mapper.toJSON(map)
assert expected == result
}
}

With this code there is quite a bit of assumptions though:

  • it will only work for Map
  • it assumes Strings are used as Map Keys
  • it will convert Maps, Lists and Objects
  • when it meets Object it will call .toString() on it to get it’s value
  • it will try to format it with tabs and new lines a bit, so it’s more pleasant for the eye and human readable

Hope it would be useful for you. Greg

TestNG NG, Next Generation of a runner for TestNG

I’ve been working with TestNG for 2 years now. The team that I work on made a decision to switch to TestNG thanks to one very important feature, @BeforeSuite and @BeforeClass. We are using Gradle as our build tool. Gradle supports TestNG tests execution.

With all the great features that TestNG comes with it also comes with features that could obfuscate test code readability. We also observed that there is no guarantee that the tests in the same class file will be executed together as a set.

With 25 minutes time of my train ride to/from work and determination to build something sweet and simple, I decided to create a Gradle plugin that will run TestNG tests in deterministic order, supporting only a small set of TestNG features.

I’ve made the code available here: https://bitbucket.org/gigu/testngng. The code is still work in progress, however there is already a functional Gradle plugin that can produce same style of XML reports as TestNG itself. There is also option of a Html report with simple and pretty style. I’ve written it in Groovy as I like Groovy. It’s Open and available to anyone.

Features supported by TestNGNG

  1. TestNGNG will recursively scan class folder passed in as parameter, in search of possible tests files. It treats this top-level folder a Suite.
  2. TestNGNG ENSURES all the tests in a class file are executed as a set of tests.
  3. TestNGNG will build a tree of tests and dependencies at the beginning of a run, before it executes a single test. The test tree is build in a form of a: One Test Suite -> Many Test Classes -> Many Tests.
  4. TestNGNG supports original TestNG annotations, it doesn’t have it’s own annotations as it is only a runner.
  5. TestNGNG support @Test annotation of a method or a class.
  6. It supports dependencies between test methods with:
    @Test(dependsOnMethods=”foo”)
  7.  It supports @BeforeSuite/Class/Test/Method and @AfterSuite/Class/Test/Method. However, @BeforeTest, @AfterTest and @BeforeMethod, @AfterMethod are treated in the same way and executed before/after each test. Just to avoid (or add to) confusion.
  8. It supports disabling of a test by enabled attribute of an annotation:
    @Test(enabled=false)

    I’m not proud of this feature though and am tempted to remove it.

  9. It supports exception expectation by expectedException attribute:
    @Test(expectedException=WhatevaException.class)
  10. TestNGNG supports data providers, however they have to be declared within the same Test Class file. Data providers could be named, or anonymous. For example:
    @Test(dataProvider = "makeMeSomeData")
      public void testWithProvider(String v1, String v2){
      System.out.println(String.format("%s - %s ", v1, v2));
    }
    @DataProvider
    public Object[][] makeMeSomeData() {
      return new Object[][]{
        {"some1", "Some2"},
        {"some3", "some4"}
       };
    }
    
  11. TestNGNG supports setup methods inherited from base classes. For example:
    public abstract class BaseForTestWithTestSetupMethods {
        public String baseValue = "";
        @BeforeMethod
        public void executeBeforeSuite() {
            baseValue = "BeforeMethod";
        }
    }
    
    public class TestClassExtendingFromBaseClass extends BaseForTestWithTestSetupMethods {
        @Test
        public void shouldPass() {
            assertThat(baseValue, is("BeforeMethod"));
        }
    }
    

Gradle plugin

Gradle plugin is very simple to use. It only requires plugin jar on a class path and TestNG as a testCompile time dependency. The plugin adds testngng task to the project. Sample use of a plugin:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath fileTree(dir: '../gradle-plugin/build/libs/', include: '*.jar')
        classpath fileTree(dir: '../testngng/build/libs/', include: '*.jar')
        classpath group: 'org.codehaus.groovy', name: 'groovy-all', version: '1.8.0'
    }
}
apply plugin: 'java'
apply plugin: 'testngng'
repositories {
    mavenCentral()
}
dependencies {
    testCompile('org.testng:testng:6.5.1')
}

Most of the other TestNG features are not covered as they are useless and very often stand in a way of tests readability and simplicity.

Don’t ask what you test framework can do for you, ask what can you do for your test framework.

How to build the plugin and use it

You would need to checkout the project from a BitBucket and build it with Gradle by typing from command line:

gradle jar

The build jar will be used by test runner Gradle plugin inside inside the testngng_gradle_sample project. You can ran gradle command from the sample project folder:

gradle testngng

I would love to see someone having a go and providing me with some feedback. There is still a whole lot of stuff on my list that needs development in the near future, like a build tests running feedback, more plugin configurations and multiple suites per project/module.

Cheers, Greg

Unit testing Grails controllers with duplicate form submission check functionality

I’ve been doing some Grail 2.0.1 development recently. I like the maturity of framework and the ease of doing things.

One of the things Grails comes with is a simple way of avoiding duplicate form submission. Have a look at the code bellow:

def myControllerMethod(){
  withForm{
    render “theGoodStuff”
  } .invalidToken {
    render “theBadStuff”
  }
}

That’s the controller bit. In your view you need to enable use of that feature by passing useToken parameter to form tag:

<g:form action=”myControllerMethod” useToken=”true”></g:form>

It looks very simple and elegant. However when we would like to test the controller we need to make sure we match the token when calling the method.

Documentation on testing Grails application and this particular functionality contains the way of doing so, however I found it not working with Grails 2.0.1.  Not much was blogged about it so I looked through the mailing lists. I found one trail and a bug report for this issue.

Anyway, to make it work, the piece of documentation from Grails, version 1.4.x explains how to do it, and it works.

In controller test method we need to place this code:

…
def token = SynchronizerTokensHolder.store(session)
params[SynchronizerTokensHolder.TOKEN_URI] = '/myController/myControllerMethod’
params[SynchronizerTokensHolder.TOKEN_KEY] = token.generateToken(params[SynchronizerTokensHolder.TOKEN_URI])

controller.myControllerMethod()
…

Happy testing. Greg

Useful links:

http://grails.1312388.n4.nabble.com/grails-2-0-testing-controller-with-withForm-invalid-notation-td4316150.html
http://jira.grails.org/browse/GRAILS-8504
http://grails.org/doc/1.4.x/guide/9.%20Testing.html
http://grails.org

Introduction to Gradle

For last few weeks I’ve been moving rather large collection of Ant, Shell and other scripts into Gradle. In my opinion Gradle is great. It has two things that I value the most when it comes to tools:

  1. The build scripts are CODE. As scripts are CODE I can do everything I could do when I code. It gives me freedom from XML
  2. It has dependencies management powered by Ivy.

Scripts are Code

The build scripts are Groovy code. Groovy is dynamic language that runs on JVM (similar to JRuby in some sense).  This means that I can test my build scripts with unit tests. This also means that I can do loops and conditions easier without weird, artificial XML tags.

Dependency management by Ivy

Gradle team, instead of inventing new dependency management, actually decided to use Ivy. Simple and easy tool that does what it says on a tin, nothing more. More important it will not try to download the Internet every time it runs.

The beginnings

First of all, if you would like to follow me with this article, grab yourself installation of Gradle. It is self-contained zip, and the only thing that you would need is Java installed.  Verify your Gradle installation by typing gradle –v from your command line.

Gradle expects your build scripts to be in a project folder in a file called build.gradle. You can write your scripts in another file but every time you execute Gradle you will have to specify where your build script is: gradle –b file_name.gradle.

To execute a task defined in build.gradle file you need to type: gradle taskName.

To see the list of all available tasks in for project type gradle tasks from command line.

Write you some tasks

Gradle has a concept of a task. Task is combined of multiple actions. One can define a task as new and assign some actions to it, or use predefined Types of task (Copy, Exec, etc.).

Example of new task definition:

task doStuff << {
  // Hey hey, I’m doing stuff
  println ‘Doing stuff’
}

Example of task that copies files:

task copySomeThings(type: Copy) {
  from ‘source’
  into ‘destination’
}

There is a very subtle difference between those two ways of defining tasks, and I got caught many times by it. The first task, doStuff doesn’t have predefined functionality. Calling method task doStuff creates task with name doStuff. However, I need to define some actions. This is the role of operator << (shift left). What actually happens is:

  1. Task doStuff is created
  2. Action that is defined in closure (between two curlies, {}) is appended to the end of the actions list of this task.  First and only action that will happen will be printed out into STDOUT.

If I need another action to follow the thirst one I defined I could do:

doStuff << {
  // Wow, now I’m doing another action
  println ‘Weheeeey’
}

The definition of copy task however doesn’t contain << assignment. It does contain the closure though. This closure is passed into the task Copy to configure it. from and into are methods defined on task type Copy that can be used to specify where to pick up files from and where to drop them. Every predefined task has some parameters and methods that I can specify. Please refer to documentation  (link to Gradle task list) for details of task.

This is important

The most common mistake I kept on making was defining a custom task and forgetting the << operator. The task did nothing and I was spending countless hours investigating why.

task doStuff(){
  // this will not happen
}

Task dependencies

In Gradle I can define dependencies between tasks. This means that when I want to execute specific task and there is another one that needs to be executed before it, Gradle will execute it too. The way to define task dependencies:

task doStuff() << {
  // I’m doing stuff
}

task doingEvenMore(<strong>dependsOn: doStuff</strong>) << {
  // Doing more
}

You can also specify name of the task instead of the task reference. This allows for kind of a lazy evaluation of build script.

task doingEventMore(dependsOn: ‘doStuff’) << { }

If you need to add dependency for already defined task you can use dependsOn method of a task.

doingEvenMore.dependsOn someOtherTask

Or dependsOn property of a task:

doingEvenMore.dependsOn = [doStuff, someOtherTask]

There is a catch with the last method: it overrides all previously created dependencies.

Then, there was Java

As of now I was describing how to create tasks and declare dependencies between them. To actually use it on a project we will need some more action. That’s why it is time to introduce Gradle Java plugin.

apply plugin:’java’

Java plugin adds some task to your build. You can check them out by executing gradle tasks from command line. Tasks that can compile, test and even jar your project.

What we are interested in are dependencies that project might need and the way Gradle handles them. By dependencies I mean third party libraries like: log4j, junit, etc. You can see dependencies in your project by executing gradle dependencies from command line.

Dependencies

Java plugin adds few dependency configurations. I’m interested in compile time dependencies, as it is needed for my project to compile and run.

apply plugin:’java”

repositories {
  mavenCentral()
}

dependencies {
  compile group: ‘log4j’, name:’log4j’, version: ‘1.2.9’
}

Inside repositories block I used method mavenCentral(). This is method predefined by Java plugin providing access to Maven central repository. As I mentioned earlier Gradle uses Ivy to resolve dependencies. Ivy in turn works rather well with Maven repositories.

Lines above will add single log4j dependency to mine project. The cool thing about it is that it will also resolve all its dependencies.

We can also add multiple dependencies by doing:

dependencies{
  compile(
    [group:’log4j’, name:’log4j’, version:’1.2.9’],
    [group:’xfire’, name:’xfire’, version:’1.2.6]
  )
}

This will add two dependencies to my project.

There are few more dependencies (runtime, compile, testCompile, testRuntime, archive) added by Java plugin. Dependencies are nothing more than a list of libraries (files) included in a build lifecycle (compile, test etc).

Configurations

One can define its own dependency configuration by creating new entry in configurations block of a build script.

configurations {
  gregWasHere
}

dependencies{
  gregWasHere group: ‘log4j’, name:’log4j’, version: ‘1.2.9’
}

Configurations are nice and simple way of adding compiletime or runtime libraries to any build. For example I used it when executing external Java process that needs extra libraries in classpath.

task executeSomeJava(type: JavaExec){
  main = “com.some.main.Bla”
  args = [“1”, “2”]
  classpath = configurations.gregWasHere
}

Summary

Gradle is really great and if you are still stuck in the world of XML and hate it, give it a try. Few tips I written in this article, could be used as a good starting point. I also found that coding to task convention that Gradle uses, simplifies the build and release process.

Few good resources to read when starting with gradle:

Mixture of technology at Google App Engine (GAE)

For some time now I am developing my little hobby project. It is called F1Dashboard and it contains informations related to Formula 1 (my favorite motor-sport).

F1 Dashboard
F1 Dashboard

Since the site is quite stable now I will share with you some technologies and ideas I used.

Platform

F1Dashboard is build and deployed on Google App Engine (GAE). I started this as an experiment and as a toy project just to learn something new.

GAE is not a new idea, it is another platform for cloud computing. Main difference from others is the Database model. GAE is using a thing called Data Store (Big Table), which is not a normal relational database that anyone used to. There is no schema defined, and querying is impossible without indexing it before. It is something like a massive bag that could store stuff in.

GAE is offered as Python and Java. The later was released long time after Python. It has limited version of JVM environment and a whole bunch of Google services to use. Lots of standard JDK types got repackaged by Google just to limit standard Java functionality, like spawning a Threads.

GAE offers standard Java servlets as web technology. This means that you could use some of the well established frameworks out there. I decided to create one myself of course 🙂 Something simple and not complicated.

Language

I choose Groovy, just because I wonted to learn another dynamic language and I liked the way Groovy did closures.

Patterns

The framework I did follows MVC pattern. It has single servlet entry. Every requested URL is parsed internally and resolved into Controller and Action. I’m using String Template as rendering engine for my views.

All database operations are entirely hidden behind Repositories, and I made it simple to switch from Objectify to Data Nucleus to anything else that comes.

I choose Objectify for its simplicity. If you are starting with GAE I would very much recommend it.

UI

The design is simple and was done by fried of mine Mike from Rosslyn Digital.

I plugged in jQuery Tools for Overlay effect. It also has a nice plugin for picture gallery.

I also added automatic Twitter updates for every message that is updated on a site.

Updates

Site is automatically harvesting for new content from various of Formula 1 related sites. There is also content from Twitter, YouTube and Flickr. It is utilizing GAE cron jobs for that.

I’m trying to gather some ideas for what I can do with the site in near future. If you have an idea, drop me an email. You can also see latest updates on a site at http://blog.f1dashboard.com

Cheers, Greg

Percent encoding in Groovy

I was writing some helpers for OAuth Twitter authorization. One of the problem I got was the encoding. OAuth is using UTF-8 and percent encoding (special style of URL ecoding).

I couldn’t find anything build-in in Java or Groovy so I wrote a very short little method that does it.

def encodeString(def stringToEncode){

def reservedCaracters = [32:1, 33:1, 42:1, 34:1, 39:1, 40:1, 41:1, 59:1, 58:1, 64:1, 38:1, 61:1, 43:1, 36:1, 33:1, 47:1, 63:1, 37:1, 91:1, 93:1, 35:1]

def encoded =  stringToEncode.collect { letter ->
reservedCaracters[(int)letter] ? “%” +Integer.toHexString((int)letter).toString().toUpperCase() : letter
}
return encoded.join(“”)
}

If you ever need something similar, use it 😉

Greg

F1 Dashboard released

In last few months I was working on a little personal project. When Google App Engine team announced that preview of Java version is available I decided to give it a try. When I was looking for a subject of my application, one of the ideas started to emerge more that others. I was looking for a web site that could aggregate all the information about Formula 1 racing. Because there was none available (or my Google search ended on first few results that were not satisfactory) I decided to make one.

Few weeks and cups of coffee latter F1Dashboard.com is released.

F1 Dashboard screen shot
F1 Dashboard screen shot

Generalities

So, in few words, F1 Dashboard is agregator of news, and media feeds from world of Formula 1. I decided not to collect content, just store information where to find it and short description.

News are harvested from a number of web sites. Twitter updates serves as another source of news.

Images are comming from Flickr, videos from YouTube.

All updates  are happening every few minutes and are 100% automated.

Technicalities

GAE supports Groovy as one of JVM programming languages. I decided to give it a try. I love Groovy, it is somewhere in between a friendly and known Java API and Ruby programming language.

It took me a moment to use to BigTable type of DataStore. It restrictes a way I used to work with databases normaly. Google provide entire environment for local development, whitch means I don’t have to upload to a cloud to see working result.

I created a little homegrown Model Views Presenter/Controller framework and used StringTemplate as rendering engine.

jQuery is the library of choice to handle all JavaScript.

Page styling was done by a friendly and kind designer from Circa82 Michael Austin (thanks buddy).

Come on guys, give it a try and tell me what you think 😉 Feedback greatlly appriciated. http://www.f1dashboard.com

Cheerios, Greg

Groovy with JDO on Google App Engine, enhancing with DataNucleus

I have some time this days and I’m trying to fly some clouds with Google App Engine (GAE), Now when they have released Java support I can try to use Groovy. Today I tried to implement some simple storage using JDO. GAE team supports some of its features via  DataNucleus. DataNucleus is using a post compilation hook to enhance any persistable classes.

Having Groovy in a lib folder of your war (GAE runs your application from this folder) I need to put asm library as well. Latest version of Groovy (1.6.1) has a dependency on asm library in version 2.2.3. On the other hand DataNucleus enhancer has dependency on asm library in version 3.1.

GAE Java SDK is shipped with Ant task for enhancing persistable classes. It also contains Ant macro definition file that makes it simple for usage. Unfortunately this macro includes all libraries from application lib folder. This is causing Enhancer to use asm library in a version that causes it to fail massively.

[enhance] SEVERE: An error occured for ClassEnhancer "ASM" when trying to call the method "org.datanucleus.enhancer.as
m.ASMClassEnhancer" on class "getClassNameForFileName" : org.objectweb.asm.ClassReader.accept(Lorg/objectweb/asm/ClassVi
sitor;I)V

Number of this error messages is as big as the number of classes in my application.

Solution is to exclude asm library from the macro that Google shipped. Easiest way is to edit, or copy to your local application folder and use it there. The line problematic line of code is in classpath element.

<classpath>
          <pathelement path="${appengine.tools.classpath}"/>
          <pathelement path="@{war}/WEB-INF/classes"/>
          <fileset dir="@{war}/WEB-INF/lib" includes="*.jar">
          </fileset>
</classpath>

The easiest way to fix it is to exclude asm library in fileset element, just like this:

<classpath>
          <pathelement path="${appengine.tools.classpath}"/>
          <pathelement path="@{war}/WEB-INF/classes"/>
          <fileset dir="@{war}/WEB-INF/lib" includes="*.jar" excludes="asm*.jar">
          </fileset>
</classpath>

Hope that helps anyone with similar problem.

Cheers, Greg