Author: Greg

9 years in The Bank – lessons learned

I worked at RBS for almost a decade. During my nine years at the bank, I got the opportunity to work with very talented and smart people, and I learned a lot. I want to use this opportunity to summarise some learnings and experiences that I encountered during that time.

Large Organisations are like Big Ships

Large Organisations, like traditional banks, are like big ships, hard to get going, require an enormous amount of energy to move and a long distance to stop. When it finally departs, it’s impossible for it to quickly and swiftly change the direction. The organization is on the constant lookout for problems far into the horizon, to be ready for a direction change maneuver. Finally, if unexpected happens, just like the Titanic, the ship will not be able to avoid the obstacle and will crash into it. Make a mental note as this will impede decisions and choices you’ll have to make.

white cruise ship

 

Smart people do dumb things – the fallacy of targets

I’ve seen people make irrational decisions, sacrificing Software quality and Customer Experience for the sake of delivering to deadlines. Deadlines quite often self-imposed, making no commercial reason. Those decisions were driven by localized targets, imposed by line managers. Values that the organization pursued, the good of customer was forgotten because the way performance was assessed, was against the targets.
The result was a poor quality software, delivered at a high cost, and at the end had to be scrapped and re-written. Beware of personal targets as those are often in direct opposition to organization vision and objectives.

Smart people do smart things – if you give them some autonomy

I’ve seen and been working in the hugely successful teams. Teams were focused, creative in solving problems and very productive. Motivation was high as well as team satisfaction. What was common for those teams was the autonomy in solving a clearly articulated problem they were given.
When you hire smart people, give them problems to solve, guide them in organization intricacies and help get the resources they need to solve problems but don’t tell them what to do (don’t micromanage).

Shared services for technology don’t work

Creating a team of specialists (for example DataPower developers), co-locating them in one place and offering their skills as services to delivery teams via the ticketing system is a recipe for disaster. Something that makes sense from the accounting point of view doesn’t work for agile teams that are building software. Again, localized targets for Shared Service teams are never aligned with the vision for product teams.

Beware of re-inventing the wheel

I’ve seen the countless amount of open-source frameworks and tools wrapped in a “thin” wrapper, offering the “same” functionality as the original. In reality, central platform teams started with good intention but hardly ever had the time, resources or expertise to deliver the promise. Avoid re-inventing the wheel, instead of wrapping try to integrate and use extension mechanisms.

gray multi spoke wheel leaning on wall

Management and Leadership are two different things

Leaders inspire people to take actions; managers help people to overcome obstacles and deal with distribution of work. It is possible to be a great leader but at the same time a terrible manager, and the other way around as well. During the nine years, I’ve not met a single Great Leader who was a great manager as well. I believe that in a large organization this happens due to a simple lack of time to perform both roles well. Beware of giving Leaders the responsibilities of managers. They might do a mediocre job at both instead of great at one.

Distributed work is hard

Distributing work on a single product creates communication overhead. For example, scaling one team into two teams creates the illusion of doubling productivity. In reality communication, teams synchronization and integration overhead will only yield a fraction of improvement. Introducing new teams, distributed across different location only amplifies the problem reducing improvements further. When scaling for improved productivity, first look into augmenting existing team and fixing all issues in the delivery process to achieve a smooth flow of work.

Working with vendors require much care

Agencies and Consultancies like to deliver software or services on their terms. The typical sales pitch will mention how beneficial it is for the customer and how productive they can be. The only problem is that quite often after the engagement is done, resulting work doesn’t integrate, is build with incompatible or problematic technologies or lacks on quality. When choosing vendors, make sure to have a set of requirements ready, including technologies to use, ways of working (e.g., integrate early and often), engineering practices and quality standards. If you hear questions related to the above list from the vendor, it’s a good sign, but don’t take anything for granted.

Scaling teams to early could backfire

This point is the expansion of the previous mark of Distributing work. If appropriate software architecture, engineering practices, and delivery pipeline are not in place, the resulting product will lack in quality and cohesive solution. Multiple teams will try to isolate itself via designing architecture and solutions that make it possible for them to work in parallel in relative isolation. This time it’s a Convey’s law applicable on the micro level. If you need to move faster, first try to optimize your delivery process before looking into scaling.

Technology is hardly ever a problem

Majority of issues that arise during software development have very little to do with technology and are associated with people relationship and communication problems. Bad communication leads to missed requirements and invalid assumptions. Make sure to validate all the assumptions early and keep revisiting information you gathered. When you have all the requirements, keep on going back to the Product Owner to confirm that what you are building is the right thing. More importantly, if it is still needed.

Finally

Above are just some of the lessons I learned. I will use this knowledge at the new organization I just started work. I think that the above might provoke some thinking for you as well as re-affirm the lessons learned for me.

Start With Why: How Great Leaders Inspire Everyone to Take Action

A couple of months ago I finished reading the book “Start With Why: How Great Leaders Inspire Everyone to Take Action” by Simon Sinek. Simon’s presentation at TED in 2009 was viewed more than 41 million times. He was invited to consult on Leadership for Microsoft and the United Nations.

SWW-Cover-High-Res

I would like to share with you a few observations and thoughts after reading the book.

Manipulation vs Inspiration

Both manipulation and inspiration are methods of making someone take action. Manipulation creates short term result. For example, if a company decides to use manipulation in their sales and marketing approach, like a price reduction, the result will be a temporary increase in sales but it will not build brand loyalty. Try to remember last time when the reduced price of a product made you become loyal to a brand.

On the opposite site of manipulation is the inspiration. Brands that inspire have a loyal group of followers who will always buy a product of said brand. This happens as the followers associate themselves with the values that represent a specific brand.

When it comes to leaders in organizations and companies, those who inspire will benefit from loyal and hard-working employees. So how does one inspire?

The Golden Circle

Simon explains that inspirational leaders and organizations all act and communicate in the same way. He calls that pattern of communication, The Golden Circle.

Golden-Circle

What it means is that communication happens from the inside out, starting with answering the “Why?” question first. Why leaders do what they do and why organizations make what they make. An answer to this simple question communicates the reason for actions, it demonstrates what someone believes in.

People get inspired by others who believe in the same things.

Biological reaction to Why?

As it turns out, humans have a very strong need to feel that they belong. It is one of the most powerful feelings, feeling based on gut. We feel that we would like to belong to a group that shares the same believes as we do as. The feeling of belonging makes us feel safe. We are drawn to organizations and leaders that are good at explaining what they believe in.

The part of the brain responsible for emotions and feelings is called a Limbic Brain. That is not the same part of the brain that is responsible for language. That is why the gut feeling, the feeling of “Just right” is hard to dress in words and explain.

Great organizations are built on the foundations of The Golden Circle and look like a Cone.

GoldenCircle

  • The Why? element of the cone includes the leader who sets the vision.
  • Larger, the How? element of the cone contains the next level of senior executives, inspired by Leader, people who know how to bring the vision to life
  • Finally, the largest part of the organization are the people who bring the vision to life by building the What?

Summary

The book itself is a great, thought-provoking read that I would recommend for anyone. It does explain how and why people are drawn to certain organizations and leaders.

Inspirational organizations and leaders know the answer to the question “why?” and they clearly communicate their beliefs through their actions. Inspiration creates the long-lasting effect of loyal followers or employees.

If you find yourself not having enough time to read the book, have a look at Simon’s TED presentation. It focuses on the core ideas of the book.

Fixing DevOps

I recently posted a one-liner on LinkedIn, that attracted a great deal of interest and thought-provoking discussion.

If I was paid a £1 for every consultancy, company or private contractors who claim to come in and “fix DevOps” for us and then fail, I would be a very rich man 🙂

In light of the comments and queries, I decided to expand on what I mean by Fixing DevOps and failing at it. First, let me start by explaining what was the trigger to write the line.

Devops toolchain

One Cheeky Email

As a Head of Software Engineering I have been targeted  by Sales representatives attempting to sell software products and software development services for quite some time now; a few days ago I received yet another email promising to Fix all the DevOps headaches we might have and change our company to become a DevOps Nirvana if only we would to bring them in.

I have been working in the financial sector for 9 years and witnessed a number of times promises that hardly ever been delivered on. I know that my industry colleagues have had similar experiences.

Thus, the above one-liner shared on LinkedIn, was born.

Some problems of large organizations

Historically, the organization I work for had nothing to do with technology. Banks offered financial services for centuries without the use of Software. Computer systems and software were adopted in the 60ties. The technology was used as an aid to business, means of making money easier and faster. Today banks would not exist without IT systems. There is more virtual money in the economy than tangible assets.

In many large organizations, technology is still looked at as an afterthought, the necessary evil that has to be dealt with in the most cost-effective way possible. Latest advances and innovation are hard to introduce. New technologies and processes are adopted at a much slower pace than technology focused organizations like Google or Amazon.

Large and complex organizations can’t exist without modern technology as well as technology makes no sense without their core business. The truth is, both sides have to work together but in reality, the way organizations are constructed prevents it from happening.

Technology is siloed into one organizational unit and business into another, each with its respective leader. Technology becomes a service organization for business. Local goals emerge, driven by local targets resulting in both organization pulling into different directions and the customer finding little to no improvement.

Let’s reiterate some of the DevOps principles at this point:

  • Focus on delivering value to a user
  • Thinking big picture – End-to-End product delivery, from inception to delivery and beyond
  • Never-ending feedback loop on the product, it’s quality and behavior in production
  • Cross-functional and autonomous teams
  • Ruthless automation of everything

#BuzzWords to the rescue

I observed the following pattern in the history of DevOps adoption with the involvement of technology leaders at different organizational levels.

A Leader hears a ‘buzzword’: DevOps. Next steps are:

  • some research into benefits,
  • multiple visits by DevOps consultancies, referring to case studies within a similar large organization,
  • a consultancy (or few) comes in to perform DevOps assessment,
  • a report is produced with information about organizational challenges,
  • recommendations on how to change and what tools to adopt

Tools become the focus area of proposed improvements as organizational changes are too problematic for A Leader. Consultancy begins the new engagement, ramping up resources and bringing in new tools. The process of “fixing the DevOps” in the organization starts.

A Leader chooses a small area of Organization to roll out new approached and tools. Neither the consultancy nor small area of Organization has enough remit nor possibility to influence any organizational changes, resulting in: consultancies automating a few basic processes, leaving behind a large backlog of future/unfinished work and a hefty bill.

Small, local improvements make little impact in the large organization. The experiment is deemed as a failure and adoption stops (until next Leader arrives or a good sales strategy from different consultancy).

Summary

Many DevOps consultancies are selling The Dream, utilizing template case studies based on the size of targeted organizations, rather than being tailored to individual requirements of said organizations. Challenges posed by organization structure in DevOps adoption are overlooked during sales negotiations. Resulting engagement creates an invalid perception of DevOps as not being fit for purpose, causing more damage than good.

The truth is that for any change to be successful, creating long-lasting effect the initiative has to come out from within, driven by ‘outside the box’ approach.

Let it go

I started my career in software development 30 years ago.

I was 9 years old when my dad bought me my first computer. In post-communistic Poland it didn’t come by easy. After weeks of going through local paper adds, he managed to secure one from a post graduate student. I became a proud owner of Atari 65XE.

As it was just a computer with no persistent storage, I had no other choice but to type in programmes myself. Initially I was copying listings for games and other software from Computer Magazines. After a short while I picked up a book on Basic and learned my first programming language.

Fast-forward few years later I finished my Master Degree in Computer Science and started my first job as a Software Developer. I progressed my career through variety of industries and technologies. At some point, I started to lead a team of developers. It was the first time when I encountered the important learning: becoming a leader means that I needed to “Let it go” . The “It” depends on context and leadership level, which I will come back to later. As well as discovering that I had to “Let things go” I also discovered that I was not very good at it.

Early days

When I started as a young tech lead on a team, I fell into following traps: trying to write code 100% of the time and making all the technical decisions myself.

Coding 100% had undesired side-effect, I was left with no time to think about software architecture and technical vision.

Similarly, trying to make all the technical decision left me becoming a bottle-neck, stretch my time thin, resulting in demotivating the team.

Fast-forward, few years later I was responsible for multiple teams and fell into similar traps: being involved in all the technical decision making, while still trying to write some code and being a developer.

As a result I had no time to write code and became unreliable team member who couldn’t dedicate time towards product delivery. I was also even bigger bottle-neck as now I was stopping multiple teams in progressing. Finally, I was not contributing towards wider organisation architecture and technology strategy as I had no time left for it.

However, recognising an issue is the beginning of a “therapy”. After reading an excellent series of Pat Kua’s articles on Leadership I discovered that I need to “Let go” of my current approach.

But … letting go is never, EVER, easy.

Change

Why is it so hard for humans to accept change and “Let it go”?

It’s because our brains are pre-programmed with recognising patterns. Early human had to fight to survive. Every change in the environment meant unexpected consequences, at time resulting in death. Diverting from common patterns causes stress. We don’t like stress and that is why we are staying in a comfort zone of what we know, as we’ve been doing it before. It feels safe.

Image result for change

How to let go

Starting point is to accept that the change is inevitable. Trying to do all the same things as before plus dealing with new responsibilities is not going to work. You will drop the ball on one of them sooner rather than later.

Next step is to identify what you need to change and “Let go of”. For me change in the tech lead role meant initially that I had to stop coding all the time, learn how to delegate, reserve time to think and visualise technical vision.

After moving to another leadership level I had to learn the language of the business, how to communicate well and delegate even more.

Most importantly I had to accept the fact that I will no longer be the Subject-Matter-Expert in all aspects of the Platform, Products and Architecture but my new responsibilities include driving the vision forward.

I found that talking with my manager about ideas and changes I was introducing helped me a lot. That is why finding yourself a mentor or a coach is helpful during the process.

Finally, there are things you can do to train you brain to accept change easier, it involves cognitive training. Things like brain games, learning new languages or taking up a task that is not in your comfort zone. All of those activities will flex your brain muscles, helping you adjust to new circumstances introduced by change.

Conclusion

Wether it’s leading the team or organisation, leadership is about vision. Organically grown leaders are often falling into traps of sticking to their previous behaviours and patterns, sacrificing their time and forgetting about the vision.

We need to “Let go” of the old habits and accept the change, resulting in a clear vision for team/organisation to follow and more productive and efficient teams.

Controlling central heating with Arduino and Raspberry Pi

When Arduino and Raspberry Pi released first versions, I did buy one of each. Being a gadget man and … well, a man, I played for a bit with my new toys and left them in a drawer.

Time has passed and not much happened with any of them. As I became increasingly unhappy with the central heating controller in my current house, I decided to take the Ino and Pi out of the drawer and actually build central heating controller that I would be happy with.

The design phase

First requirement of the new heating control system was the least intrusive installation as possible.

The dial thermostat that I have at home, works as a simple switch: switching the heating on when temperature falls bellow pre-set; and switching heating off when temperature rise above another pre-set. I decided to use this simplicity in my design. All I needed to do is hook with one 230V cable into that thermostat (230V, hell yeah).

With entry point sorted, I went into the controller bit. Arduino board will control the Relay, which will simply switch heating on/off. For the temperature reading I chose the Digital Thermometer which offered more stable reading than the Analog one (included in Arduino Uno starter kit). I’ve also added LED to indicate when the heating was on.

Leveraging serial port used to program Arduino, I decided that the board will send temperature updates and receive setup command from Pi via USB connection. It will also be powered via the same. This also solved a problem of 2 separate power adapters for both Arduino and Raspberry Pi.

The build phase

I’ve been trying few things before arriving with the below solution.

Electronics

The Arduino circuit is very simple.

circuit-smaller

Three elements connected to Arduino with a couple of resistors, not much.

The LED is not necessary, it’s there to indicate when the heating is switched on.

The final prototype doesn’t look very attractive, but the lot is hidden under the furniture and the only elements sticking out are the temperature sensors and a bit of LED.

Final prototype of my controller

 

 

Software

I had to write three separate pieces of software:

Arduino

For the temperature sensor I included two extra libraries, the OneWire protocol library and a DallasTemperature sensor library. I use 0.5 centi-degree approximation of a temperature reading.

Temperature reads are sent via Serial Port on every loop. Arduino also expects the Float number on a Serial Port. The received number indicates the desired room temperature for Arduino.

To limit the sensor reading fluctuations, the control of relay changes after at least 10 successful consecutive reads of the same temperature from the sensor.

Raspberry Pi

The software that runs on Raspberry Pi does the following:

  • It waits on the temperature updates from Arduino, and stores the Updates in Memory (for the latest update) and in simple file based H2 database (for historical data),
  • It exposes REST API for UI to get the temperature information and receive the new settings,
  • It schedules temperature changes according to schedule stored in the JSON file.

I started the code in Python, but it was running slow. I did a simple comparison of execution time for Prime number algorithms, and Java 8 was beating Python. On a single core Raspberry Pi 1, it was a good incentive to change the platform. I chose Kotlin programming language as it was new to me and I wanted to learn it.

As a framework for Event driven application I choose Vert.x 3. For Serial Port communication, a bit dated RXTX library.

The UI / Phone controller app

The Web App I build works on computers as well as mobile devices. I did choose React as a UI framework with Material UI components. The lot is build with Webpack into a small set of html/js files.

The testing phase

The testing phase involved connecting everything together, starting it, and hoping that I will not sense the smell of burning electrics and experience no explosions. In other words, a standard scientific and engineering approach 🙂 .

I do run the setup for 4 weeks continuously and it not failed so far.

Summary

This is the first time I used my skills to build something that interacts with physical world. It gave me a great feeling of achievement and satisfaction. I know that I could buy something that looks much nicer and probably works better but I learned a lot during the process.

Raspberry Pi 3 was released while I was building my design and I switched to it. You can see it in the pictures. I also want to switch Arduino Uno prototyping board to Arduino Nano.

All the code and more detailed technical description is available for you to grab from my GitHub repository on https://github.com/greggigon/Home-Temperature-Controller .

5 key learnings from the Innovation Spike

Innovation

Last week I took part in the innovation spike at RBS. This post is my personal retrospective of the spike, findings and lessons learned.

Outline:

  • What is an innovation spike?
  • Format of the spike
  • Technologies and tools used
  • Findings and lessons for the future

What is an innovation spike?

In RBS the innovation spike means an event where small team works in a startup mode, trying to deliver new product or a service by using new technologies. The reasons for the spike are to explore new technologies as well as prove technical possibility and produce proof of concept for business.

The prototype that we delivered was functional enough to spawn discussions around business model and to take a look at product from the user perspective. It was also small enough to simply throw away if business assumptions doesn’t hold or technology is not good enough.

It is possible there is a better name for this event, however “Innovation spike” sounds great and naming things is hard.

Format of the spike

Our small team had 6 developers, graphics designer and a product owner. As a team we had all the technical and non-technical skills required to work on every aspect of product development.

We did prepared a little bit for the spike. Our product owner had a pretty good idea of business that we would like to validate. We had a small infrastructure prepared with some basic tools and did read on the choice of technology for the spike.

The spike took place in The Bakery London, which is a great place for a startup gigs with fantastic people ready to jump on problem and help. We run it for 4 days (Tuesday to Friday) with one day of preparation (Monday) and demo to business on the last day.

Technologies and tools used

There was a number of candidate technologies we wanted to try during the spike. We ended up with the following:

  • Microsoft Azure for our infrastructure needs. Ubuntu VMs with Docker for spike tools and for our applications deployment
  • GitLab, Let’s – Chat, Taiga.IO, Jenkins, Docker, Docker registry with Front-end and ApacheDS as LDAP authentication was our development tools stack.
  • Meteor framework for all front-end and some backend development and Vert.x 3 with Java as a backend API services.
  • VirtualBox with Vagrant for consistent development VMs.
  • Atom, Visual Studio Code and IntelliJ as development IDEs

Some of the technologies were familiar to us, some were entirely new. We picked the above technologies as we wanted to give them a try and provide feedback to the rest of the bank.

At the end we didn’t use Taiga.IO at all as good old fashioned boards with stickies worked great.

Findings and lessons learned

The bellow findings represent my thoughts after holding conversations with others in the team.

1. Location is important

Being away from the office in a different location changes how you think. I mean it. Many things adds up towards it. Lack of dress code and non corporate style of office made everyone feel relaxed. Food and drinks took away the need of thinking about those things during the day. Little technical touches made us being more productive. Things like fast and easy Wireless Internet, big flatscreen connected to Apple TV so that everyone could share their screen when needed.

IMG_20150728_135347~2

2. Cross functional, co-located team is important

It was great for all of us to sit around one big table, where we could pair with each other, ask each other questions or simply showcase what we’ve just done. Business (or product) owner could provide constant feedback and get us on the right path. Spike team in turn could immediately report back what was technical doable or what would take more time than we had.

The range of different skills across our team meant that we could handle all aspects of delivery lifecycle: design, development, testing and release.

There was simply no time wasting and no communication issues.

3. Having a leader is important

Having a leader is important as we got constantly refocused on the goal and next piece of work to handle.

A leader is not there to manage! Role of a leader is to propose initial structure and adapt it as we go along. Innovation spike is no time and place for a project manager.

In our case, our Mighty Leader prepared everything for the spike, time, place and the right people. On the first day of spike he introduced initial structure. We had a session with product owner, than we jumped to a small design phase and we started coding. In average, once every 3 hours we stopped and synched on what we managed to do.

4. Coming prepared is important

As I mentioned earlier the innovation spike is about exploring new technologies. It’s a learning exercise. However, we did prepare a bit and here’s why. Having a good starting point saves time.

Not everyone have to prepare on every aspects of the spike. I took time to learn a bit about Azure Cloud and prepare some basic infrastructure for our team. Others did complete introduction and tutorials on Meteor framework. Someone else created Mockup APIs. We all learned from each other. Someone always had an answer to a problem others were facing. Being prepared is always good.

5. Simple and good enough is GOOD … ENOUGH

Simple and good enough is more than enough for the innovation spike because it’s easier and faster to make. At the end, the final product is something that could be simply thrown away or taken as a proof of concept for a bigger project.

We were constantly reminded about not paying to much attention to unnecessary details, but to focus on the business functionality and pushing boundary of the tested technology.

Summary

Innovation spike was great learning experience. Finding out about new technologies and business opportunities. It was also great fun to be on. The key to its success was location, cross-functional team and being prepared. Having a good and focused leader made us very productive and good enough approach caused us to deliver successful and working proof of concept in a very short time.

What is happening with My Personal Kanban

Some time has passed since my last blog post. I have been working on quite a few things recently. My Personal Kanban is one of them.

I did run a short survey between some of the My Personal Kanban users, asking them about things they would like to see in future releases. Some of the most important include:

  • seamless synchronisation with Cloud across many devices
  • mobile version of the application
  • tagging of cards that makes them searchable
  • Master Board with cards across all Kanbans

At this moment I’m heavily working on the development of the Synchronisation with Cloud. It will replace rather unfriendly process of saving and loading from Cloud. I’m buried in rather complex algorithms for resolving Conflicts between Kanbans.

I’m also re-architecting and modularising Client side services, so they could be reusable in more than one version (same services to be used in Web version and Mobile version).

One of the major changes would involve delivery or deployment mechanism. The application itself will no longer be available as downloadable ZIP file, but a static website that will work offline, Chrome App and Firefox Extension and Mobile HTML5 application for both iOS and Android platforms.

Kanban is also getting a major UI redesign with help from my friend and a great designer Mike.

I am still juggling 9 to 5 day job, preparing to present on a conference and attempting to finish “Principles of Reactive Programming” on Coursera. That is the main reason why new release of My Personal Kanban is taking longer.

Path to work

The road is long but there is light at the end of a tunnel. Stay tuned.

New My Personal Kanban 0.8.0 released – upgrades and new features

I’ve released new version of My Personal Kanban, browser based Kanban board.Kanban column

There are new features as well as simple upgrades to libraries, etc. I also removed Bootstrap UI library.

New features in the latest 0.8.0 release include:

  • Per column limits – possibility to setup limit of cards in the column. If the limit is reached, it’s impossible to add or drag more cards into column.
  • No restrictions to the number of columns. Previously you could only create Kanbans with 3,4,6 and 8 columns. Now it is possible to create Kanban board with any number of columns between 2 and 10.
  • It is also possible to add and remove columns via a column settings menu.

Future plans for My Personal Kanban will include auto-sync with cloud and some major changes in the cloud sync protocol. I need to make those changes before I start development of a mobile version of MPK.

I do hope that new features come in handy and that you will keep on using MPK as your personal Kanban board.

My Personal Kanban version 0.7.0 released

Kanban columnI’ve released a new version of My Personal Kanban. My Personal Kanban is a very simple in-browser Kanban Board application. It is designed to work with no Internet connection, persisting content in a modern browser’s data store. MPK can also store your Kanban encrypted in the Cloud with full data privacy.

New features follow closely previously delivered functionality extending into the specific requests by some MPK users raised on GitHub.

New features in the latest 0.7.0 release include:
• Importing previously exported Kanban from text (JSON) file. It’s a follow up on Export functionality from previous version.
• Change colour of a Column. This functionality comes with a new Column settings button.
• Possibility to select existing Kanban as a Template for New Kanban. If there is a specific structure, column names and colours that you like, you can reuse the setup when creating new Kanban.
• Each Kanban has a unique URL in the Browser address bar, which makes it possible to open or bookmark specific Kanban (this change forced me to introduce Angular.js router, a bit of info for devs).

As well as changes in the latest 0.7.0 release, there are also:
• Updates of libraries to latest versions
• Bugs fixes

As I’m getting closer to release of version 1.0.0, My Personal Kanban is getting future complete. Some of the new functionality that will come before final release includes:
• Pomodoro timer
• Blocked section in the columns
• Import/Export to CSV file

I’m also planning a Mobile version to follow on both iOS and Android platforms.

I would love to hear from you if you are using My Personal Kanban, in what way and what functionality is missing.

Greg

My approach to JSONP limitations

Why JSONP?

Kanban columnDuring the development of My Personal Kanban I stumbled across interesting problem. One of My Personal Kanban features that I was developing, was the possibility to upload Kanban to Google Cloud (Google App Engine application).

My Personal Kanban is designed to work off the local file system, without the need of Internet connection. It means that trying to send something into Interweb is going to hit modern Browser security settings.

The browser security feature prevents web request from being made to a site with a different domain. It also stops from making a request to anything other the same file if you have opened any web application from the local file system. This kind of behaviour is default across any modern web browser.

Fortunately, there is a special type of requests browser will allow happening and it’s JSONP. It’s a GET request with a callback parameter. Callback is a name of the function that the browser will call when it receives successful response from the web.

HTTP GET request limits

My Personal Kanban is written in JavaScript with Angular; server side on Google App Engine is a very simple Servlet written in Groovy.

When I finished my first implementation and started to test it locally with Google App Engine SDK it all looked good. However, upload stopped working when I tested with a real GAE deployment.

Quick research confirmed that different web server implementation might have different settings for Maximum Length of HTTP GET parameters. Those GET parameters are used to send JSONP request data to the web server. Google App Engine has different limit than GAE SDK (which uses Jetty).

I also discovered that the upload was working in one browser but stopped working in another. As it turns out, browsers have its own limits of length of HTTP GET request. It is literally the length of the URL you will put in your browsers address box.

My Personal Kanban is sending a bit of data. It’s not Megabytes, however still too much for the HTTP GET parameters.

Choices of workaround

I thought of writing My Personal Kanban as Chrome extensions. It would enable me to overcome the JSONP limitations however it would bind my application to work on Chrome only. So I ditched the idea.

I decided to chop data into small chunks and send it to the server. To minimize errors of the transfer I invented this Client-to-Server protocol:

  1. All calls during transmission to server are in order, so the data can be assembled in correct way.
  2. First a handshake is made announcing the beginning of transmission for specific user, with information of how many data chunks will be sent.
  3. Chunks of data are sent only if previous chunk was sent successfully.
  4. Finally MD5 hash of data is sent so the server can verify that what was received is correct.

On server side I decided to store chunks for a user in session which in case of Google App Engine is stored in Memcache backed by Data Store. What happens on the server:

  1. Server receives a handshake and creates new Array for data chunks (or removes previous one if transfer was not completed or not successful).
  2.  Server receives data chunks and places them in the Chunk Array stored in session.
  3. When server receives Kanban hash it concatenates the Array in order into String and validates against received hash. If the hash is valid it stores the Kanban in the Data Store.

Details and code samples

The uploaded Kanban is Encrypted with user key and kept on the Cloud encrypted to ensure data privacy. I’ve chosen Rabbit as encryption algorithm. As a side effect to encryption, transmitted data doesn’t need to be encoded and sanitized.
For encryption I’m using fantastic CryptoJS library. CryptoJS also includes MD5 hash implementation.


md5Hash : function(stringToHash){
	return CryptoJS.MD5(stringToHash).toString();
},

encrypt: function(stringToEncrypt, encryptionKey){
	var utfEncoded = CryptoJS.enc.Utf8.parse(stringToEncrypt);
	return CryptoJS.Rabbit.encrypt(utfEncoded, encryptionKey).toString();
},

decrypt: function(stringToDecrypt, encryptionKey){
	var notYetUtf8 = CryptoJS.Rabbit.decrypt(stringToDecrypt, encryptionKey);
	return CryptoJS.enc.Utf8.stringify(notYetUtf8);
}

Thanks to fantastic Angular Promise API it is very easy to implement ordered transmission of data.

var encryptetKanban = cryptoService.encrypt(kanban, this.settings.encryptionKey);
var kanbanInChunks = splitSlice(encryptetKanban, 1000);

var promise = sendStart(kanbanInChunks.length);
angular.forEach(kanbanInChunks, function(value, index){
	promise = promise.then(function(){
		return sendChunk(value, index + 1);
	});
});

return promise.then(function(){
	return checkKanbanValidity(encryptetKanban);
});

My Personal Kanban is Open Source project and available for browsing at https://github.com/greggigon/my-personal-kanban/

Conclusion

The above approach made it possible to upload data to the server from any browser eliminating issues related to default browser’s security settings.
Unfortunately it comes with a cost of more HTTP requests and custom server side coding.

For small amount of text data it is good enough for me, perhaps it would be good enough for you. Let me know your thoughts.