My Personal Kanban version 0.7.0 released

Kanban columnI’ve released a new version of My Personal Kanban. My Personal Kanban is a very simple in-browser Kanban Board application. It is designed to work with no Internet connection, persisting content in a modern browser’s data store. MPK can also store your Kanban encrypted in the Cloud with full data privacy.

New features follow closely previously delivered functionality extending into the specific requests by some MPK users raised on GitHub.

New features in the latest 0.7.0 release include:
• Importing previously exported Kanban from text (JSON) file. It’s a follow up on Export functionality from previous version.
• Change colour of a Column. This functionality comes with a new Column settings button.
• Possibility to select existing Kanban as a Template for New Kanban. If there is a specific structure, column names and colours that you like, you can reuse the setup when creating new Kanban.
• Each Kanban has a unique URL in the Browser address bar, which makes it possible to open or bookmark specific Kanban (this change forced me to introduce Angular.js router, a bit of info for devs).

As well as changes in the latest 0.7.0 release, there are also:
• Updates of libraries to latest versions
• Bugs fixes

As I’m getting closer to release of version 1.0.0, My Personal Kanban is getting future complete. Some of the new functionality that will come before final release includes:
• Pomodoro timer
• Blocked section in the columns
• Import/Export to CSV file

I’m also planning a Mobile version to follow on both iOS and Android platforms.

I would love to hear from you if you are using My Personal Kanban, in what way and what functionality is missing.

Greg

My approach to JSONP limitations

Why JSONP?

Kanban columnDuring the development of My Personal Kanban I stumbled across interesting problem. One of My Personal Kanban features that I was developing, was the possibility to upload Kanban to Google Cloud (Google App Engine application).

My Personal Kanban is designed to work off the local file system, without the need of Internet connection. It means that trying to send something into Interweb is going to hit modern Browser security settings.

The browser security feature prevents web request from being made to a site with a different domain. It also stops from making a request to anything other the same file if you have opened any web application from the local file system. This kind of behaviour is default across any modern web browser.

Fortunately, there is a special type of requests browser will allow happening and it’s JSONP. It’s a GET request with a callback parameter. Callback is a name of the function that the browser will call when it receives successful response from the web.

HTTP GET request limits

My Personal Kanban is written in JavaScript with Angular; server side on Google App Engine is a very simple Servlet written in Groovy.

When I finished my first implementation and started to test it locally with Google App Engine SDK it all looked good. However, upload stopped working when I tested with a real GAE deployment.

Quick research confirmed that different web server implementation might have different settings for Maximum Length of HTTP GET parameters. Those GET parameters are used to send JSONP request data to the web server. Google App Engine has different limit than GAE SDK (which uses Jetty).

I also discovered that the upload was working in one browser but stopped working in another. As it turns out, browsers have its own limits of length of HTTP GET request. It is literally the length of the URL you will put in your browsers address box.

My Personal Kanban is sending a bit of data. It’s not Megabytes, however still too much for the HTTP GET parameters.

Choices of workaround

I thought of writing My Personal Kanban as Chrome extensions. It would enable me to overcome the JSONP limitations however it would bind my application to work on Chrome only. So I ditched the idea.

I decided to chop data into small chunks and send it to the server. To minimize errors of the transfer I invented this Client-to-Server protocol:

  1. All calls during transmission to server are in order, so the data can be assembled in correct way.
  2. First a handshake is made announcing the beginning of transmission for specific user, with information of how many data chunks will be sent.
  3. Chunks of data are sent only if previous chunk was sent successfully.
  4. Finally MD5 hash of data is sent so the server can verify that what was received is correct.

On server side I decided to store chunks for a user in session which in case of Google App Engine is stored in Memcache backed by Data Store. What happens on the server:

  1. Server receives a handshake and creates new Array for data chunks (or removes previous one if transfer was not completed or not successful).
  2.  Server receives data chunks and places them in the Chunk Array stored in session.
  3. When server receives Kanban hash it concatenates the Array in order into String and validates against received hash. If the hash is valid it stores the Kanban in the Data Store.

Details and code samples

The uploaded Kanban is Encrypted with user key and kept on the Cloud encrypted to ensure data privacy. I’ve chosen Rabbit as encryption algorithm. As a side effect to encryption, transmitted data doesn’t need to be encoded and sanitized.
For encryption I’m using fantastic CryptoJS library. CryptoJS also includes MD5 hash implementation.


md5Hash : function(stringToHash){
	return CryptoJS.MD5(stringToHash).toString();
},

encrypt: function(stringToEncrypt, encryptionKey){
	var utfEncoded = CryptoJS.enc.Utf8.parse(stringToEncrypt);
	return CryptoJS.Rabbit.encrypt(utfEncoded, encryptionKey).toString();
},

decrypt: function(stringToDecrypt, encryptionKey){
	var notYetUtf8 = CryptoJS.Rabbit.decrypt(stringToDecrypt, encryptionKey);
	return CryptoJS.enc.Utf8.stringify(notYetUtf8);
}

Thanks to fantastic Angular Promise API it is very easy to implement ordered transmission of data.

var encryptetKanban = cryptoService.encrypt(kanban, this.settings.encryptionKey);
var kanbanInChunks = splitSlice(encryptetKanban, 1000);

var promise = sendStart(kanbanInChunks.length);
angular.forEach(kanbanInChunks, function(value, index){
	promise = promise.then(function(){
		return sendChunk(value, index + 1);
	});
});

return promise.then(function(){
	return checkKanbanValidity(encryptetKanban);
});

My Personal Kanban is Open Source project and available for browsing at https://github.com/greggigon/my-personal-kanban/

Conclusion

The above approach made it possible to upload data to the server from any browser eliminating issues related to default browser’s security settings.
Unfortunately it comes with a cost of more HTTP requests and custom server side coding.

For small amount of text data it is good enough for me, perhaps it would be good enough for you. Let me know your thoughts.

My Personal Kanban – use your own local cloud

I’ve finished new release of My Personal Kanban 0.5.0 and  version of My Personal Kanban Server. New Cloud features enable to Upload and Download Kanban to any Cloud server, accessible via web.

Cloud Setup Menu

My Personal Kanban Server will accept uploads and downloads from My Personal Kanban and store it on disk. You don’t need to generate key, however you can use the same one, you use with MPK Cloud.

Details of how to install server and run it can be found here: https://github.com/greggigon/my-personal-kanban-server .

Technology involved in the MPK Server

I decided to learn Clojure and write the Server in Clojure. I thought the problem was simple to implement it while learning new programming language. I’ve picked Ring to help me. It provided just enough to handle web requests and left me with everything else to code.

Clojure is great language and I hope I will be using it more in the future.

 

My Personal Kanban – Cloud storage encryption

Not too long ago I finished working on a new release of My Personal Kanban. It is a small Open Source project that I have been working on. The aim is to provide a very simple Offline Kanban board in a browser window.

The latest release contains one major improvement for Cloud Uploads. It encrypts (ciphers) Kanban with user provided Key before uploading it into the Cloud. Thus, there is no possibility to view or decipher Content of the Kanban as the Key is not uploaded to the Cloud but only stored in your browser local cache.

The only data stored by Cloud in the readable form are: username and email address. In the wake of recent Spying and Snooping events, I believe this feature of My Personal Kanban should provide ease for everyone concerned about their privacy.

Cloud Encryption key setup

I’m using Rabbit encryption algorithm from CryptoJS library.

The latest release 0.4.0 also includes some bug fixes and code cleanup. You can grab it from here.

Once you have the new version, when you upload for the first time, the Kanban will be encrypted.

Next release that I am currently working on, will include persistence to a local Cloud storage (a small version of Cloud application running on user network and user server).

New TeamCity plugin for user authentication via Atlassian Crowd

TeamCity ScreenRecently I have finished some work on the TeamCity Crowd Plugin.

This plugin enables TeamCity to use Crowd for user authentication. It could be used in places where Crowd is used to manage access to other products (for example Atlassian JIRA or Confluence). Thanks to this plugin the same user base will be able to access TeamCity.

The plugin is open Source, available on GitHub, binary build downloadable on BinTray.

TeamCity Crowd Plugin is using Atlassian Java REST Client to connect to Crowd. More detailed description of functionality on GitHub.

Please do get in touch with feedback/improvements.

Greg

Do Less to Do More – My Personal Kanban in the Clouds

Kanban columnAs you might already know I did a small application called My Personal Kanban, that serves as offline Kanban board. It’s HTML application which stores your Personal Kanban on your device. It free to use and Open Source.

I just released a new feature and service for My Personal Kanban that allows you to Upload and Download your Personal Kanban Boards to Cloud.

The service and functionality is in Beta phase but it would be great if you could give it a try and provide me with some feedback.

Heres a video of how the Cloud features work.

Here is the link to the previous post with Themes.

If you want to help me and contribute a Theme or some code, please do get in touch.

Greg

Horizontal Scaling of Atlassian Crowd

Context

Atlassian Crowd is an identity management server for Web Applications. It integrates very well with other Atlassian tools like JIRA, Confluence, Stash, Fisheye, Crucible, etc. We also used it with some custom plugins as Authentication and Authorization mechanism for our CI Servers (TeamCity), SonarQubue (via OpenID) and Source Code Repositories (SVN, Atlassian Stash).

Crowd was connected to Active Directory with LDAP connector and delegating Authentication to it. The picture bellow represents out setup.

Crowd Setup

We had about 15000 users and 1500 groups across 3 directories. Each directory delegating authentication to Active Directory.

Performance issues

After a short while we started to get user complaining that they were unable to log in to JIRA or checkout code from Subversion, etc. Turned out that Crowd was slowing down. Profiling the system Crowd was running on showed us that there was plenty of CPU and Memory left for it.

The problem turned out to be Active Directory. Most specifically the slow response time for authentication requests into Active Directory at times of substantial load.

Plan and decision

We’ve decided that it would be nice to have a number of Crowd instances, serving different Applications.

Implementation

We’ve created Master -> Slave setup. All the users and groups’ management would happen in Master Crowd and propagate to Slave Crowd. We picked applications and grouped them by authentication load and set them up on appropriate Crowd Servers.

Crowd configuration got duplicated and catalogs imported so the Slave Crowd was still using Active Directory for authentication. Then we tackled the problem of keeping Crowd Master and Slave in Sync (users and groups).

Keeping Slave up to date

Unable to find a good out of the box solution we developed a Custom Crowd plugin and a bunch of web services.

Plugin functionality was rather simple: listen to any User or Groups events in Crowd (add/update/remove) and perform a simple HTTP get request with change details to preconfigured URL.

We’ve created two types of web services:

  1. The Web Services for the Crowd Plugin to call when the change happened. It was asynchronously calling to next Web Service that performed the changes in Slave Crowd.
  2. The Web Service performing changes in Slave Crowd.

The picture below represents the solution.

Crowd Simple Setup

Multiple Slaves

We’ve ended up having multiple Crowd Slave instances. One of the instances had to be older and different version.  Separating the Web Services gave us possibility for using different versions of Crowd Java Client Libraries.

We have also used this setup to keep our UAT environment with up to date data from Master Crowd.

The Web Services are stateless and have no database.

Crowd Full setup

Full Crowd synchronization

As additional functionality we implemented full synchronization triggered from WS 1. WS 1 takes list of all the users and groups for each Directory from Master Crowd. Once collected it will call each WS 2 with full update.

This functionality makes it very easy to bring new UAT/DEV Crowd environments and populate it with Production data. It also makes it possible to Sync entire directory if one of the Web Services goes down.

Summary

The solution described above gave us horizontal scalability and possibility of working with different Crowd Client libraries and Crowd Versions. It also makes it easy to upgrade Crowd Instances.