My approach to JSONP limitations

Why JSONP?

During the development of My Personal Kanban I stumbled across interesting problem. One of My Personal Kanban features that I was developing, was the possibility to upload Kanban to Google Cloud (Google App Engine application).

My Personal Kanban is designed to work off the local file system, without the need of Internet connection. It means that trying to send something into Interweb is going to hit modern Browser security settings.

The browser security feature prevents web request from being made to a site with a different domain. It also stops from making a request to anything other the same file if you have opened any web application from the local file system. This kind of behaviour is default across any modern web browser.

Fortunately, there is a special type of requests browser will allow happening and it’s JSONP. It’s a GET request with a callback parameter. Callback is a name of the function that the browser will call when it receives successful response from the web.

HTTP GET request limits

My Personal Kanban is written in JavaScript with Angular; server side on Google App Engine is a very simple Servlet written in Groovy.

When I finished my first implementation and started to test it locally with Google App Engine SDK it all looked good. However, upload stopped working when I tested with a real GAE deployment.

Quick research confirmed that different web server implementation might have different settings for Maximum Length of HTTP GET parameters. Those GET parameters are used to send JSONP request data to the web server. Google App Engine has different limit than GAE SDK (which uses Jetty).

I also discovered that the upload was working in one browser but stopped working in another. As it turns out, browsers have its own limits of length of HTTP GET request. It is literally the length of the URL you will put in your browsers address box.

My Personal Kanban is sending a bit of data. It’s not Megabytes, however still too much for the HTTP GET parameters.

Choices of workaround

I thought of writing My Personal Kanban as Chrome extensions. It would enable me to overcome the JSONP limitations however it would bind my application to work on Chrome only. So I ditched the idea.

I decided to chop data into small chunks and send it to the server. To minimize errors of the transfer I invented this Client-to-Server protocol:

1. All calls during transmission to server are in order, so the data can be assembled in correct way.
2. First a handshake is made announcing the beginning of transmission for specific user, with information of how many data chunks will be sent.
3. Chunks of data are sent only if previous chunk was sent successfully.
4. Finally MD5 hash of data is sent so the server can verify that what was received is correct.

On server side I decided to store chunks for a user in session which in case of Google App Engine is stored in Memcache backed by Data Store. What happens on the server:

1. Server receives a handshake and creates new Array for data chunks (or removes previous one if transfer was not completed or not successful).
2.  Server receives data chunks and places them in the Chunk Array stored in session.
3. When server receives Kanban hash it concatenates the Array in order into String and validates against received hash. If the hash is valid it stores the Kanban in the Data Store.

Details and code samples

The uploaded Kanban is Encrypted with user key and kept on the Cloud encrypted to ensure data privacy. I’ve chosen Rabbit as encryption algorithm. As a side effect to encryption, transmitted data doesn’t need to be encoded and sanitized.
For encryption I’m using fantastic CryptoJS library. CryptoJS also includes MD5 hash implementation.

md5Hash : function(stringToHash){
	return CryptoJS.MD5(stringToHash).toString();
},

encrypt: function(stringToEncrypt, encryptionKey){
	var utfEncoded = CryptoJS.enc.Utf8.parse(stringToEncrypt);
	return CryptoJS.Rabbit.encrypt(utfEncoded, encryptionKey).toString();
},

decrypt: function(stringToDecrypt, encryptionKey){
	var notYetUtf8 = CryptoJS.Rabbit.decrypt(stringToDecrypt, encryptionKey);
	return CryptoJS.enc.Utf8.stringify(notYetUtf8);
}

Thanks to fantastic Angular Promise API it is very easy to implement ordered transmission of data.

var encryptetKanban = cryptoService.encrypt(kanban, this.settings.encryptionKey);
var kanbanInChunks = splitSlice(encryptetKanban, 1000);

var promise = sendStart(kanbanInChunks.length);
angular.forEach(kanbanInChunks, function(value, index){
	promise = promise.then(function(){
		return sendChunk(value, index + 1);
	});
});

return promise.then(function(){
	return checkKanbanValidity(encryptetKanban);
});

My Personal Kanban is Open Source project and available for browsing at https://github.com/greggigon/my-personal-kanban/

Conclusion

The above approach made it possible to upload data to the server from any browser eliminating issues related to default browser’s security settings.
Unfortunately it comes with a cost of more HTTP requests and custom server side coding.

For small amount of text data it is good enough for me, perhaps it would be good enough for you. Let me know your thoughts.