Functional Programming in JavaScript

Last week I was really surprised to find Functional JavaScript: Introducing Functional Programming with Underscore.js by Michael Fogus in the local library and I just finished it and wanted to leave a short review here.

This book really delivers what the title promises: an introduction to functional programming in JavaScript using the library Underscore.js. It doesn't teach you JavaScript nor Underscore.js but teaches what the different functional programming concepts are and how they can be implemented in JavaScript. This is done in less then 250 pages of densely, in a good way, written text and example code. Starting from the basics like first-class functions, applicative programming, variable scoping and closures, the book moves on to higher-order functions, currying and partial function application. Then some side steps are made with a great chapter on recursion which ends with the trampoline and  a chapter on other important functional aspects like purity and immutability. Next is a chapter about flow-based programming, what it is, why it matters and different ways to define flows in your programs. The last chapter makes the connection with object-oriented programming and introduces mixins.

I really enjoyed reading this book because it is written very fluently without heavy (unnecessary) jargon and probably at a sweet spot on my learning curve. I've already read Real World Functional Programming: With Examples in F# and C# by Thomas Petricek and Jon Skeet and the first chapters of SICP but I haven't used it a lot in the wild. I've written my share of JavaScript programs but nothing very advanced, except maybe a Google Maps like library from scratch. If you're new to JavaScript AND functional programming then would advice against this book but otherwise, if you're motivated and don't let you get scared away by the first chapters then everything will be fine. But some playing around with the examples (like I did in this fiddle) and learning the basics of how to call passed in functions and how the often used Underscore.js functions (map, reduce, ...) work might be needed to get the most of this book. Overall this book is a very complete introduction to functional programming, the only thing I missed was a part on functional pattern matching. Note that this book is more about what introducing different functional programming techniques then about when and how to apply this techniques in your day-to-day programming.

Other books you might be interested in:
JavaScript: The Good Parts by Douglas Crockford (he popularized JSON and wrote JSLint and JSMin)
JavaScript: The Definitive Guide by David Flanagan
JavaScript Allongé by Reginald Braithwaite
Real World Functional Programming: With Examples in F# and C# by Thomas Petricek and Jon Skeet
Learn you a Haskell for Great Good! by Miran Lipovača
Learn you some Erlang for Great Good! by Fred Hébert

Creating a node.js GeometryService : part 3

Its been a while since I posted my first and second post on a GeometryService based on node.js and PostGIS.

Today I want to announce that I open sourced my work in progress. It is in a pre-alpha state and it can be found here. The only thing it can do at the moment is convert ESRI Json geometries from one coordinate system to another. This is done by converting the geometry to WKT, creating a sql statement for the re-projection and then converting the GeoJSON result back to ESRI Json.

A sample input url when you started the geometryservice is : http://127.0.0.1:3000/rest/services/Geometry/GeometryServer/project?inSR=4326&outSR=3819&geometries={%22geometryType%22:%22esriGeometryPoint%22,%22geometries%22:[{%22x%22:-117,%22y%22:34},%20{%22x%22:-115,%22y%22:25}]}. This returns

{"geometries":[{"x":-115.00495449693217,"y":24.99533030379053},{"x":-117.00524224953409,"y":33.99507554185914}]}

as a result. The intermediate sql statement looks like this :

SELECT st_asgeojson(st_transform(st_geomfromtext(geometries.geometry, '4326'), '3819')) g FROM  (SELECT ST_ASTEXT('POINT(-117 34)') geometry UNION SELECT ST_ASTEXT('POINT(-115 25)')) geometries

.

Most of the logic of the code happens in geometryService.js. The conversion from and to ESRI Json happens in datatransformer.js. As stated in my previous post I also created some tests for the functions in the datatransformer.js. There is still a lot to do, especially everything related to error handling and logging and off course the implementation of the other geometry operations.

The main dependency is a runnning instance of PostgreSQL with PostGIS. As the connection string is not configurable yet you'll have to change it in the executeSQL function in geometryService.js.

The code can be found on following url https://bitbucket.org/gissolved/node_geometryservice and it is licensed under the MIT license.

If you like this then consider buying me a book or a license for Sublime Text 2.

Visualizing Geometries

In this post I'm going to give some more information about the GeometryVisualizer that I created recently.

The goal of the GeometryVisualizer is to be able to quickly visualize a GeoJson, WKT or ESRI JSON geometry. I created this to be able to quickly see how some ESRI JSON and WKT geometries looked like and visually verify some geometry conversion code I'm writing for my GeometryService in node.js. You can also use this to visualize the WKT or GeoJSON output of a PostGIS geometry query.

The ESRI JSON geometries are rendered with the ArcGIS JavaScript API. The WKT and GeoJSON geometries are rendered with OpenLayers.

Some code

In this section I'll show some parts of the code that made this web application possible and compare the ArcGIS JavaScript API with the OpenLayers API. First we'll take a look at the initialization of the map. The most notable facts about the initialization are that we need to add a layer before we are able to add graphics and that disabling the zoomwheel is less straight forward with the OpenLayers API.

ESRI JavaScript API map initialization:

var map = new esri.Map("esriJsonMap");
map.disableScrollWheelZoom();

// add a layer and directly hide it because otherwise we can't create graphics
var basemapURL= "http://server.arcgisonline.com/ArcGIS/rest/services/World_Street_Map/MapServer"

var basemap = new esri.layers.ArcGISDynamicMapServiceLayer(basemapURL);
map.addLayer(basemap);
basemap.visible = false;

OpenLayers map initialization:

var options = {
  maxExtent: new OpenLayers.Bounds(-1000000000, -1000000000, 1000000000, 1000000000),
}
var map = new OpenLayers.Map('openLayersMap', options);

var controls = map.getControlsByClass('OpenLayers.Control.Navigation'); 
for(var i = 0; i<controls.length; ++i){ 
    controls[i].disableZoomWheel(); 
}

var layer = new OpenLayers.Layer.ArcGIS93Rest( "ArcGIS World Street Map", 
                "http://server.arcgisonline.com/ArcGIS/rest/services/World_Street_Map/MapServer",
                {layers: '0'} );
map.addLayer(layer);
layer.setVisibility(false);

To be able to render an ESRI JSON geometry string with the ArcGIS JavaScript API you have to do some specific steps :

• Clear the graphics
• Parse the geometry string representation
• Create a graphic and set its symbol according to the geometry type.
• Update the extent of the map

Not that for updating the extent, I had to access an undocumented property of the graphics object and for point graphics the extent had to be enlarged.

function addGeometry(geometryText) {
    
    if(map.graphics === null) return; // map not loaded yet or something else is wrong
            
    map.graphics.clear();

    var geometryJson = JSON.parse(geometryText);
    var graphic = new esri.Graphic(geometryJson);

    if (!graphic.symbol){
      switch(graphic.geometry.type){
        case "point":
        case "multipoint":
          graphic.setSymbol(new esri.symbol.SimpleMarkerSymbol());
          break;
        case "polyline":
          graphic.setSymbol(new esri.symbol.SimpleLineSymbol());
          break;
        default:
          graphic.setSymbol(new esri.symbol.SimpleFillSymbol());
          break;
      }
    }

    map.graphics.add(graphic);
    // use undocumented object
    var ext = graphic._extent;
    // if point expand the extent
    if(ext.getWidth() < 0.00000001 || ext.getHeight() < 0.00000001){
      var factor = 1;
      ext.update(ext.xmin - factor, ext.ymin - factor, ext.xmax + factor, ext.ymax + factor, ext.spatialReference);
    }

    map.setExtent(ext.expand(2));
}

The process of adding WKT and GeoJSON geometries was very similar.

• Remove all features
• Parse the geometry string representation with a predefined formatter (OpenLayers.Format.WKT or OpenLayers.Format.GeoJSON)
• Update the extent of the map.

Note that I didn't had to specify the symbology and zooming to the extent of the feature was easier. The WKT parsing was very straight forward. For the GeoJSON parsing I had to add two things. First I had to ensure that the parsed JSON had a property "type" with as value "Feature" and the parsed object returned a feature collection with one feature instead of directly returning the feature.

function addGeometry(geometryInput, parser, type){
  vectorLayer.removeAllFeatures();
  var feature = parser.read(geometryInput);
  vectorLayer.addFeatures(feature);
  if(dojo.isArray(feature)){
    feature = feature[0];
  }
  var bounds = feature.geometry.getBounds();
  bounds = bounds.scale(1.1);
  map.zoomToExtent(bounds);  
}

function addWkt(geometryText) {
  var wktFormat = new OpenLayers.Format.WKT();
  addGeometry(geometryText, wktFormat);
}

function addGeoJson(geometryText) {
  var geoJsonFormat = new OpenLayers.Format.GeoJSON();
  var geometry = JSON.parse(geometryText);
  geometry["type"] = "Feature";
  addGeometry(geometry, geoJsonFormat);
}

This are the most interesting parts of the code. The full source code is in the GeometryVisualizer.

Any questions/remarks/improvements ? Let me know !

Creating a node.js GeometryService : part 2

This is my second post on what I learned from creating an implementation of the geometryservice specification in node.js. The first post can be found here.

By default node.js doesn't reload your files when they have changed but while developing this can be very handy. The most easy to use tool for node.js on windows that monitors your files for changes, with an easy way to set the node.js debugging flag, is nodemonw. You can download nodemonw at https://github.com/cenanozen/nodemonw. Once you've downloaded the executable I suggest to copy it to your %Appdata%\npm directory or another directory thats in your path. To start nodemonw I now run the following command nodemonw --debug index.js. In the next section it will become clear why I added the --debug flag.

To be able to debug my node.js application I installed node-inspector (npm install -g node-inspector). As you can read in the readme of node-inspector it is really easy to get started with node-inspector. You just have to start node-inspector and then open http://127.0.0.1:8080/debug?port=5858 in your favorite WebKit based browser. On Wikipedia I found this list of WebKit based browsers. The most well known ones for Windows are Google Chrome and Safari. A screencast on node-inspector can be found here. There is also a node-inspector playlist on YouTube.

To be able to test parts of my GeometryService I used vows (npm install -g vows). Vows is an asynchronous behavior driven development framework. More info about it can be found on http://vowsjs.org/. I am now going to show a small part of the code from my GeometryService and some tests I wrote for this code. My directory structure for the code I'll show looks like this: lib/
-- datatransformer.js
test/
-- datatransformers.test.js
In datatransformer.js I started a function to convert an ESRI geometry JSON object to wkt based on its geometry type.

exports.esriGeoJsonToWKT = function esriGeoJsonToWKT (geometryType, geometry) {
  if(geometryType === "esriGeometryPoint") {
    return "POINT(" + geometry.x + " " + geometry.y + ")";
  }
}

And the content of datatransformer.test.js is:

var dt = require("../lib/datatransformer");
var vows = require('vows');
var assert = require('assert');

vows.describe('Esri geometry JSON to WKT').addBatch({
  'when converting esriGeometryPoint {"x":-117,"y":34}':{
    topic: function(){ 
      var geomType = 'esriGeometryPoint';
      var p = JSON.parse('{"x":-117,"y":34}');
      return dt.esriGeoJsonToWKT(geomType, p);
    },
    'we get "POINT(-117 34)"': function(topic){
      assert.equal(topic, "POINT(-117 34)");
    }
  },
  'but when converting esriGeometryPoint {"x":-117.01,"y":34.02}':{
    topic: function(){ 
      var geomType = 'esriGeometryPoint';
      var p = JSON.parse('{"x":-117.01,"y":34.02}');
      return dt.esriGeoJsonToWKT(geomType, p);
    },
    'we get "POINT(-117.01 34.02)"': function(topic){
      assert.equal(topic, "POINT(-117.01 34.02)");
    }
  }
}).exportTo(module);

As you can see I created two tests for converting point geometries from ESRI JSON to wkt. One for integer coordinates and one for decimal coordinates. The easiest way to run this tests with vows on windows was opening a commandline in the root directory of my node.js project and type vows --spec. This will run the tests it finds in the test and spec directories of your project. My output looks like this :

? Esri geometry JSON to WKT

when converting esriGeometryPoint {"x":-117,"y":34}
V we get "POINT(-117 "4)"
but when converting esriGeometryPoint {"x":-117.01,"y":34.02}
V we get "POINT(-117.01 34.02)"

V OK » 2 honored (0.007s)


On a side note if you ever encounter that node can't find any of your globally installed packages then it might help to add a new User Variable called NODE_PATH with as value %AppData%\npm\node_modules. That was it for this post more posts on this project will follow.

As I announced in this post, I open sourced the code of this project. It can be found in this bitbucket repository.

Creating a node.js GeometryService : part 1

Ever since I encountered the geoservices-rest-specification (pdf) I've been thinking about creating my own implementation. I also wanted wanted to try out node.js so I combined both ideas and started implementing a GeometryService in node.js. If you've never heard of node.js, this is the short introduction from the homepage of node.js:
Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices. The GeometryService is a web service that contains GIS related utility methods like project, intersect, buffer,... A sample server can be found here http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Geometry/GeometryServer.

I first downloaded and installed the windows installer of node.js version 0.65. After a reboot all locations where added to my path so that I can now start a new cmd window and type node. Note that when you install packages globally with the node package manager called npm (npm install <package> -g) npm (the node package manager) then the packages are installed in Windows 7 under %AppData%\npm (C:\Users\<username>\AppData\Roaming\npm). The full list of packages is located here search.npmjs.org.

To get started learning node I first read the node.js tutorial at nodebeginner.com. By following this tutorial I got a great insight in how to create your own webserver. At this moment I'm continuing to build my webservice based on this code but I plan to use some framework like express or journey in a later stage. This is not very urgent because I'll first focus on the JSON output of the GeometryService and thus won't need to output any html.

For my first version of the GeometryService I decided to use PostGIS as my geometry processor. I know this introduces an extra overhead. But I think this is the easiest way to get something up and running in a short period of time. If you don't want to bother with installing PostgreSQL and PostGIS I suggest you to download the Community edition of the Open Geo Suite. To be able to connect to the database I installed node-postgres with following command line: npm install pg -g. If you're on windows and get build errors you might try to add a file called true.cmd to the directory where node.exe resides (e.g. C:\Program Files (x86)\nodejs) with as content exit 0. Below is a short snippet on how I connect to the database. Make sure to replace all placeholders in the connection string.

var pg = require("pg");

function executeSQL(sql, parameters, resultCallback){
  var connectionString = "pg://username:password@host:port/databasename";
  pg.connect(connectionString, function(err, client) {
    client.query(sql,parameters, function(err, result) {
      // TODO add error handling
      console.log(result);
      resultCallback(result);
    });
  });
}

That was it for today. In the next part of the series I'll write about debugging node.js and about my progress on the GeometryService.

As I announced in this post, I open sourced the code of this project. It and can be found in this bitbucket repository.

C# REPL

Until recently when you wanted to test small snippets of C# code you had to create a small console application or use the immediate window while debugging or use something like LinqPad.

But a few weeks ago Microsoft finally announced a solution for this called Roslyn. The goal of Roslyn is to provide an API for the compiler. This is still just a CTP but it is very promising. One of the features of Roslyn is an interactive window for C# also called a REPL (read eval print loop). Note that some C# features like Linq query expressions, events and the dynamic and async keywords have not been implemented yet. More information on the Roslyn CTP can be found here Introducing the Roslyn CTP and here http://msdn.com/roslyn.

Tip of the day:
When you want to reevaluate or edit a previous entry then use ALT+Up and ALT+Down.

Bonus tip:
You might also wanna try out Jash. This is a javascript shell that can be opened on any website with a bookmarklet. It features some code completion and is very practical for trying out things you`re unsure about.