Implementing a REST API with node.js

Every so often there is the necessity to implement some mock-up web services or maybe even a prototype which shall offer its functionality via web services. This happened to me once again in March 2015 when preparing the API for a hackathon. To be precise, there already was an API, but it did not really fit to the requirements of the hackathon’s sponsor. Therefore, we decided to implement a second API (as a mock API) on top of the existing one. Our, second, API should add additional functionality to the existing services.

It was quite clear that the API should be REST with JSON as transport protocol. Unfortunately, we only had less than 4 weeks to design, implement and test the mock API. Thus, only a very lightweight technology could make this possible, like node.js.

Being JavaScript on the server, node.js is indeed a very lightweight technology. In its most basic configuration it will require you to write only one JavaScript file, which is then read and executed by node. Furthermore, it come with a module concept which lets you include already existing modules into your code with the “require” statement. Node also brings a packet manager (named “npm”) which lets you easily install all those modules to your project’s file system. In our mock API we used the following imports:

var http = require('http');

var express = require('express');

var url = require( "url" );

var path = require("path");

var fs = require("fs");

var queryString = require( "querystring" );

For our purposes, the “express” module is very important: it enables us to write REST services like in the following example. This short piece of code implements a web service, using the HTTP “GET” method, which echoes all input back to the response.

var api = express();

api.get('/echo', function(req,res) {

       var myobj = "";

      

        // parses the request url

        var theUrl = url.parse( req.url );

       

        //check if there is a query       

        if ((theUrl.query != null) && (theUrl.query != "")) {

             // gets the query part of the URL and parses it creating an object

             var queryObj = queryString.parse( theUrl.query );

              

             // queryObj contains the data of the query as an object

             // and jsonData is a property of it

             myobj = JSON.parse( queryObj.jsonData );                   

       }

       res.json( myobj );

});

As this examples shows, it’s really quite simple. For the response, you have to create a JavaScript object (which also might be a quite big one with arrays included and so on…) and transform it to JSON with “res.json”. This data is then returned to the client. There is a catch, though: with a HTTP “GET” method the input parameters will be encoded in the URL and therefore there is an upper limit of data which can be input to the service (around 8 kB, depending on the browser and other factors).  So, in order to potentially input more data into the web service, a “POST” method should be used, like in the next example. This one also shows how to extract parameters from the request header and do some basic authentication. Anyway, please bear in mind, it's still only a prototype source code (e.g. there should be another return code than 400 if an error occurred).

api.post('/api/booking', function (request, response) {

     //Read header & body

     var jsonData = request.body.requestPayload;

     var requestID = request.header('requestId');

     var userName = request.header('userName');

     var userPassword = request.header('password')

     var timeStamp = request.header('timeStamp');

     if (!checkAuthWithResponse(userName, userPassword, response)) return;

     resultAsString = '';

     var err = oVali.runHeaderValidation(request);

     if (err.length > 0) {

  

          // some logging and error handling here

     } else {

          // some business logic here

     }

     Response.send( resultAsString, 400);

})

Our mock API was intended to implement additional business logic on top of the existing API. The client would then call the mock API’s services to execute this business logic. Because the logic would only be depending on the values of the input parameters of the services there was no need for additional data storage or caching. Nevertheless, it would be quite easy as well to integrate a NoSQL database into node.js, for example.

Our mock API based on node.js was implemented just in time for the Hackathon and it was used there with very good results. We experienced no outages and had no problems with the performance as well.

As a summary, what are the advantages of using node.js for REST API development? We can name the following:

1. As a developer, you are up and running very quickly. Install node, download some modules, write ONE file, and you are done.
2. You don’t need to learn a lot of tools and IDEs (low overhead).
3. The feedback cycles are very short, enabling you to develop with high speed.
4. Integration with file system and databases is easy.
5. Many standard problems can be used out-of-the-box with only a few lines of code.
6. There is a big community, helping you out if you run into problems. Online documentation and FAQ is huge.
7. It’s very easy to transport node projects. Just zip and unzip the files and you are done.
8. There are a lot of possibilities to deploy and run node projects on the web. Pricing starts very low as well.

The bottom line is: node.js is a perfect choice for implementing prototype REST services. In how far this could be extended to full blown operational services – that’s another matter.

The third age of portals

After their advent in the late nineties web portals came a long way. Nowadays, they are complex and powerful tools – which demand knowledge and experience to be utilized. To understand this, it is necessary to realize, that portals went through three ages, which were shaped by different paradigms about what portals are and what they ought to be used for.

 The first age of portals – the prehistory of web portals so to speak – began in the late nineties and went until 2006. In those years, portals were mainly used as GUI integration technology: they integrated already existing applications with one login and with a unified navigation or menu. Application integration was the main concern during that era. That’s why portals still carry around a lot of different technologies how integration can be achieved: iframe, link integration, widgets, portlets, WSRP and others. Portlets became standardized with JSR 168 and JSR 286 and seemed to open up a new world of GUI components, some people even calling it GUI SOA. 

The second age of portal started around 2007. From this time onwards, due to the necessities of web 2.0, people began using portals to implement web sites of all sizes, which put the emphasis on content. This is why most portals today offer extensive support for content integration like standardized JCR interfaces or even whole off-the-shelf web content repositories. But still, portals were regarded as an integration technology (cf. picture), albeit a complex one, offering individualized views for different roles of users. Moreover, portals became equipped with so-called social collaboration capabilities, like wiki, chat, communities and groups. Portal frameworks were very successful with this task – much more than with the integration objective – and a lot of programmers got to know them better and in more detail. Soon they realized: there is even more to portals…

Portals as integration technology

The third age of portals began around 2010: portals were also used as web application frameworks. This came in handy, since portals already are equipped with what all the web applications need: login, session management, user management, navigation, database connectivity and many many more. Portals not only offer technology but also concepts how to apply them and coordinate them. Thus, people began building business web applications based on portal frameworks, which also meant to use portlets as building blocks for these applications. This is where we are now: we are denizens of the third age.

Actually, this is where a lot of trouble begins. But that’s a different story, and maybe another post.

Should Domain Objects contain business logic?

Recently I started a discussion with another software architect about where to implement the business logic in a SOA-like business layer. There would be service classes with methods which implement service functions like "bookFlight" or "upgradeFlight". We also introduced a "model" which consists of domain objects, holding the data to be manipulated by the services. The domain objects would also be stored in and read from a database and would serve as parameters for our service, meaning that they would be transferred via JSON or XML to the service consumer. Now, the question is: where should we implement the business logic? Should all of it be in the service functions or should all logic go into the domain objects or should we have a mix?

The picture below shows two services using two domain objects each. One domain object is used by both services. Should the logic be distributed? It can get even more complicated, since domain objects may also call one another, thereby using the logic of the other domain object.

But what is business logic anyway? Validation is often cited here, but that’s not all of it. More important are algorithms, calculations, rule evaluations and transformations which are driven by business requirements. Consider a domain object “passenger” which represents the customer of an airline. The passenger might have a method “getSignificance()” which returns the value of importance that the passenger has for the airline. This significance can depend on many things, for example his frequent flyer status, his age, his flight frequency, his marital status, the people he know at the airline etc. The algorithm to calculate this value also can be very complicated and it can change frequently. The significance of the passenger is so important, that it is calculated very often and it always refers to one single passenger. Makes it sense to implement “getSignificance()” at the passenger domain object?

The other solution would be to implement it in the service, e.g. in the “bookFlight” service, which uses it to calculate the price. But what if “upgradeFlight” also needs to know the significance of the passenger? Would you just duplicate the code and put it in the other service as well? That would lead to messy maintenance problems and a lot of discussion with the QA people (and for good reasons!).

Many people argue that domain objects without business logic would lead to a “clear separation of logic and data”. But, what are we talking about? It is just that, namely the combination of data and logic in objects, what the object oriented programming once set out to achieve. Thus, logic-less domain objects would rather indicate procedural programming, leading to something old-fashioned like a transaction script which is encoded in the service methods. It really reminds me of the old C programming style, putting all data in “structs” and just weaving some procedural code around it. But after all, we want to be object-oriented, don’t we?

The term “domain object” comes from “domain model”, meaning that all these classes make up the model which represents the business and problem domain. In object oriented modeling you simply identify all the objects which occur in the business domain and then you create one class for each object type found. Of course, the business domain not only contains information (“data”) – it also contains actions and activities (“methods”). Thus, domain objects naturally should contain logic as well.

Martin Fowler once coined the term “anemic domain model”

(http://martinfowler.com/bliki/AnemicDomainModel.html) for domain objects without logic. He calls it an “anti-pattern”. And I think he is right. But on the other hand, I don’t think that all the logic always should go into domain objects.

Domain objects should contain only the business logic, which is closely related to the data of the object and to the business entity’s meaning which corresponds to the current domain object. E.g., the basic logic to upgrade a passenger should not go into the passenger domain object (Although language seems to indicate that the passenger is upgraded that is not so. The booking is upgraded.). If there is a “booking” domain object the logic belongs there or into the service if not. Also, if many other domain objects are involved in the algorithm, this kind of logic should also go rather into the service class.

I would expect a service method implementation to look something like this:

myServiceMethod() {

     domainObject do1 = dao.fetchNeededDomainObject();

     do1.doSomething();

     if (do1.getSomething()) == someValue) {

         // some logic goes here    

         domainObject do2 = dao.fetchNeededDomainObject();

         do2.doSomethingElse();

         do1.alter ();     

     }

     // maybe some more logic goes here as well

     return do1;

}

As one can see, there is business logic in the service method. It is that logic, which is needed to coordinate the work of the domain objects and to drive the overall process. The services delegate most of their work to domain objects. Or, to put it another way, only that logic, which is not clearly bound to one domain object but rather to the overall process, should be found in the service. Nevertheless, domain logic must stay separate from persistence and presentation logic. This can be achieved easily e.g. by using the DAO pattern and by adding an extra presentation layer on top of the services. .

One further point: what about using the domain objects to transfer data? They could e.g. be transformed into JSON or XML and streamed to a client. Would that indicate that we should get rid of all the logic, since it could not be transferred so easily? But, as I just described, it is not the domain object that is transferred, it is a marshaled representation of it – meaning that only the data of the domain object is transferred. If the client was a Java client, the data could be un-marshaled into another domain object on the client, which would feature its method and business logic again.

So, I am absolutely in favor of solid object oriented modeling, with a well formed domain model. And this means, that domain objects must be able to hold logic, since otherwise they are nothing more than primitive data types in disguise.

SOA Service Design - Part 3

In this third part of my discussion of SOA service design I’d like to concentrate on how to group and structure the services found.  Let’s remember: the general goal we’re trying to achieve here is to break up an “application” into smaller parts which each encapsulate clearly defined business and/or technical functionality (s. picture below). These “smaller parts” are often called services and may be deployed in an enterprise service bus (ESB) for production.

 But how do we find the services in the first place? As a short recap of the method presented in parts 1 and 2, here is a step by step walkthrough:

1. Get somebody who is very familiar with the business processes and requirements of the system in question and go through all the use cases step by step. Focus on who does what, what information does he or she need to do that and what data is needed to make decisions in the process.
2. While going through the processes model them, e.g. with the modeling technique I outlined in the first two parts of this series (the legend is shown below).
3. While modeling you will find the services as they are called by the activities. Focus on what is needed to build the user interface imagined by your business expert.
4. Sometimes you will encounter situations where your former services will not fit to new processes you just model. That’s part of the game. In these cases you have to adjust your service (e.g. take them apart or join them) in order to fit all of the situations when they are called. If you cannot make them fit, this may be a sign that a new service is needed.

Thus, we end up with a bunch of process models containing services. Now what? At the end – which is the beginning of design and implementation – we need a precise definition of our services, at least precise enough that the programming team can implement them. Therefore, the names of the services won’t do for that. We have to write a short documentation for each service. It’s good practice to group the service methods we found with our process analysis by a business related criterion. That would lead us to describe the service belonging to one group in a close context in the document. This is a good ides, because services which are close business-wise also share other common traits for example pre- and post-conditions and validation functions which should be described consistently.

I recommend grouping the service with regard to the domain mode, which is the data model which contains all of the major business object types of the processes in question.

The results can be depicted graphically which could look like the following picture:

As you can see, there can be services which belong to more than one entity (like “identify” which can be group with “passenger” and with “missing baggage”). That’s absolutely normal and can lead to even more constraints for these services, which should be documented. Now remains the question of granularity. There are clues which can be gained from the grouping: If some of your services in the group contain very concrete verbs (like “create” and “delete”) than it should be a warning sing, when others are very fuzzy or even miss verbs totally. A switch in granularity (level of detail) from one group to another can occur, when e.g. one group acts “supportive” to another. This happens in the example above, when detailed passenger data (like phone number) is retrieved via a fine grained service, while the missing baggage reports are more abstract.

Now, we arrived at the end of this short series about SOA service design. Questions are more than welcome and I will be very happy to post any relevant answers here on the blog. I’d also like to here about you personal experiences with SOA modeling, so feel free to contact me.

SOA Service Design - Part 2

In part 1, we tried to come up with a way to discover the services needed for a new application. The method we explore mainly uses the processes (or use cases) of the respective application, tries to take them apart into activities which are executed one after another. And then – looking closely at the activities – we find the services needed to get it implemented (or executed if you prefer).

Now, we will refine that approach and therefore, we will look at some more complex examples. The following process model depicts a use case called “create lost & found report”. In this process the user can enter data concerning an object found by the crew inside a plane. To help him fill in the data we need a service which gets all needed information about the current flights (flight numbers etc.). If the user is ready with the data he can hit the button “ok” and the system will create the lost & found report (a data record containing all the entered information). The activity where the report is created is depicted with white background color, meaning that this activity should be executed by the system without user interaction. This is important, for automatic activities often need information different than user-driven activities. Furthermore, the data returned by the corresponding service (e.g. the key of the new report) must be forwarded to another activity (“display”) because the automatic activity does not have a user interface. The forwarding is also depicted using an orange arrow. The GUI-less activity “print” which is always triggered after “display” does not need a new service, since it can benefit from the data already given back by the “create” service. It’s worth to note, that we differentiate between read-only and read-write services, which is shown as white or colored “lollipop” bubble, respectively.

But as you might have guessed, the processes can still become more complex. And we should now help ourselves by restricting us to the simplest use cases since that would probably hide some of the most interesting services from our analyses. The next process model below shows forks – locations where the process may take different routes depending on some decision. These forks, depicted as diamonds like in UML, are of special interest to us, since the decision which way to go nearly always depends on some data (or algorithm depending on data) which must be delivered via a service. In our syntax, we either need either a dedicated service to provide the necessary data (like “Get Configuration” in the example) or we can derive the information from the service executed by an activity closest before the fork (like “Search Report” which might deliver zero reports). This process model also shows a refinement activity (“Assign Report”), which again can contain a whole process of its own. We always use refinements like this in order to keep the readability of the model at a comfortable level. Another reason to use refinements can be to further reuse in the model itself, since “Assign Report” can be called in other process models as well. Another example of reuse – process switching – can be seen by the elliptic bubble at the top right (“controlling”) which stands for another use case. The user can decide to stop executing the current process at this point and continue in the “controlling” process instead.

As you can see, we have to enlist very sophisticated patterns of process modeling. But this is necessary if we really want to model all necessary use cases with the needed level of detail. If we wouldn’t employ refinement and process switching we would soon be lost in our processes and would not be able to continue with the real objective of our work – finding the services. Nevertheless, we are mostly done with describing the modeling technique. In the next part of this discussion we will turn to the questions left: But how can we be sure, if these services are at the right level of detail? How can we group and order them and how should they be developed and deployed?

SOA Service Design - Part 1

In recent years SOA has come out of the „trough of disillusionment”. Many companies and IT service providers now use it in earnest to strengthen reuse, agility and quality and to decrease cost. But one of the central questions for each SOA project still remains: which are the concrete services which are needed? Should some functionality be split up in two services or rather be implemented in a single one?  Whether these services will be implemented as web services, java classes or stored procedures does not really matter and can be put aside for a moment. In the following I will describe a method how to identify the relevant services and how to find the right level of detail, in short a method for service design.

According to my experience a top-down approach leads to sensible results. Foremost the services are governed by the business process which shall be implemented (or are implemented, respectively). This means, in order to find the services, we have to consider the relevant processes and how they can be subdivided in activities. Since services provide logic (functionality) and data (information), exactly these two aspects have to be considered when analyzing the processes. Thus, we also get to criteria, which help us to find the right level of detail: activities, which don’t need special data or logic, also don’t need special services. Therefore, they can be excluded from the process description. On the other hand, most processes have decision points, which govern the control flow, and which need data or logic in order to make the decision. This means, that services are needed, in order to support the control flow in the process, and thus sometimes special activities must be added to support these services.

Let us look at a simple example. The diagram shows a simple top-level process, which only has two activities: “login” and “menu”. The dotted lines which lead from the activities to the services indicate which services are called from which activity. E.g. the “login” activity needs a login service which then directly or indirectly accesses an LDAP in order to execute the login. From a process perspective the login could be subdivided even into more activities like “enter ID” and “enter password”. But since no further services would be derived from that, we stop at the current level of detail. The “menu” activity calls two services: one provides the menu entries and the other provides the current language. We don’t consider the sequence of the service calls inside one activity, because this doesn’t have any consequence for the service identification. But why are the services for the menu entries and the language separated in the first place? In this case it could indeed be handled together in one service. In our example we assume, that the language is needed in other processes as well. Therefore, we introduce a separate service, in order to further reuse.

To sum it up: when using this method the key question is: which data and which logic is needed for the current activity? Sometimes this question can be answered from a business perspective alone. But often background knowledge in system design and application architecture is necessary. Thus, service design based on business analysis alone is fraught with risk and it can lead to incomplete results.