Express Web Application Development

There is an interesting story behind the origin of Express. You will understand Express better if you know the story, so let me share the story of how Express came into being.

Sometime in February 2009, Ryan Dahl had an epiphany about combining JavaScript and Google's V8 engine to create a new system-level programming platform. He christened the platform as Node.js (Node), and released v0.0.1 in the same month.

Node was very well received by the web development community, and it started to grow very rapidly in popularity.

Apart from being a general-purpose software development platform, Node provided a web server API (Application Programming Interface), using which developers could create web apps using JavaScript as the backend programming language.

However, there was a problem with Node's web server API: It was a little too low level, and you had to write and re-write many of the web server functions in your web apps. Modularity and extensibility became a problem for any project that was even moderately big.

Within five months of Node's release, in June 2009, T.J. Holowaychuk, released an open source project named Express to make web development a little easier in Node.

Express was inspired by Ruby's Sinatra and built on top of Node's web server API. It was a little crude, but provided some of the niceties—such as a routing system, session and cookie support, MIME helpers, RESTful interface, HAML-based views, and so on—one might expect from a web development framework.

However, Express v0.0.1 was very different from what Express 3 is today. Perhaps, the only thing common in between them is the name "Express".

In June 2010, Sencha, under its Sencha Labs, started an open source project named Connect, to solve the modularity and extensibility issue in the Node web server API. The project was inspired by Ruby's Rack web server interface. Tim Caswell, a Sencha employee, and T.J. Holowaychuk, were roped in to lead the project.

Like Express, Connect was also built on top of Node's web server API, and came with a middleware system, which allowed small re-usable programs to be plugged onto it to handle HTTP-specific functionalities.

Connect middlewares took care of many of the commonly required functionalities in web apps for Node. On top of that, anyone could write their own middleware for their apps. Connect considerably improved the modularity and extensibility of the Node web server API.

By now, there were two different web development frameworks for Node: Express and Connect—one was inspired by Sinatra, and the other by Rack. This caused a bit of confusion in the Node community, especially with Holowaychuk working on both of them.

But as luck would have it, it became obvious that Express and Connect were actually complementary frameworks. So, in July 2010, Holowaychuk decided to re-architect Express to run on top of Connect, effectively merging Connect with Express to create a new incarnation of Express in v1.0.0.

With Express v1.0.0, there was no more confusion about which web development framework to choose in Node. Express was Connect with additional functionalities built on top of it. To this day it remains the same—Express continues to use the Connect middleware, and any change in Connect is invariably reflected in Express.

So, that is the story of how Express came into being and how Connect is related to it.

As an Express developer, you might rarely deal with Connect directly, but you will be using a lot of middlewares in your projects. Middlewares in Express are referred to as Express middlewares and not Connect middlewares, although technically they are Connect middlewares. You will learn more about middlewares in upcoming sections in this and the next chapter.

The application object is an instance of Express, conventionally represented by the variable named app. This is the main object of your Express app and the bulk of the functionality is built on it.

This is how you create an instance of the Express module:

var express = require('express');
var app = new express();

The following is a brief description of all the properties and methods available on the application object:

The HTTP request object is created when a client makes a request to the Express app. The object is conventionally represented by a variable named req, which contains a number of properties and methods related to the current request.

The following table lists all the properties and methods of the req object and provides a brief description of them:

The response object is created along with the request object, and is conventionally represented by a variable named res. While it may sound a little strange that both of them should be created together, it is a necessity to give all the middlewares a chance to work on the request and the response object, before passing the control to the next middleware.

The following is a table of properties and methods on the response object:

Many beginners in JavaScript get stumped while using Node for the first time because they are not familiar with asynchronous (async) JavaScript and callback functions (callbacks). Node and Express are built on the concept of async operations, so it is imperative that you understand the concept before you proceed any further.

Unlike the more common synchronous functions, asynchronous functions do not return immediately; at the same time they do not block the execution of its succeeding code. This means other tasks are not piled up waiting for the current task to be completed. However, to resume control from the async operation and to handle its result, we need to use a callback function. The callback function is passed to the async function to be executed after the async function is done with its job.

Here is an example of using a timer to illustrate how callbacks work:

var broadcast = function(msg, timeout, callback) {

  // initiate an async call using a timer
  setTimeout(function() {
    // the first message
    console.log(msg);
    // execute the callback function
    callback();
  }, timeout);
};

broadcast('Is there anybody out there?', 1000, function() {
  console.log('Message sent');
});

We passed in a callback to the broadcast function, which will be executed after the message is "broadcasted" after one second.

Though Node is synonymous with async operations, it still provides a sync alternative to many of its operations. However, it is recommended to stick to the async versions, else you will very likely lose the non-blocking advantage of Node.

A Node module is a JavaScript library that can be modularly included in Node applications using the require() function. What the module is capable of is entirely dependent on the module—it can be simple helper functions to something more complex such as a web development framework, which is what Express is.

If you have used npm to install something, you have used a node module. A lot of them are installed as command-line tools, such as the express command. A lot more of them are installed as libraries to be used with a Node program.

The bulk of web server-related functionality in Express is provided by its built-in middlewares. Features not supported by Express out of the box are implemented using Node modules.

Since Express provides just the bare minimum functionality of a web server, it does not support some common but crucial functionality, such as connecting to a database, sending e-mails, and so on. In such cases, you will need to find and install the appropriate Node modules and use them to get your task done.

The fact that Express does not come baked in with opinionated modules or methods to accomplish tasks beyond handling HTTP requests is a good thing, because it keeps the framework bloat-free and gives its users the freedom of choice to use any module or method according to their specific requirements.

The Node community is very active and has developed modules for almost every requirement on a typical web project. So remember, if you are looking to do something tricky or complex, probably there is a Node module for it already, if it does not exist, probably you should create it and share it with the Node community. If you are in no mood for sharing with others, make it a private Node module and keep it to yourself.

If it makes you wonder what is the difference between a public and a private module: public modules can be published on the npm registry and installed by the general public, whereas private modules remain private.

As you start working with Express, you will realize that writing your own modules will greatly help in modularizing your app. So, it is essential that you learn how to write them.

There are two approaches to writing Node modules: one involves attaching properties and functions to the exports object, the other involves assigning JavaScript objects to the module.exports property of a module.

The attachment to exports approach is pretty straightforward, as you can see from the following example:

var name = exports.name = 'Packt';
var secret = 'zoltan';

exports.lower = function(input) {
  return input.toLowerCase();
};

exports.upper = function(input) {
  return input.toUpperCase();
};

exports.get_name = function() {
  return name;
}

exports.get_secret = function() {
  return secret;
}

Anything attached to the exports objects is available as a public property or method of the instance of the module. Any variable defined with the var keyword and not attached to the exports object becomes a private variable of the module. Save the preceding example code in a file named mymod.js, and include it in a file named test.js with the following code:

var mod = require('./mymod.js');

console.log(mod.name);
console.log(mod.lower('APPLE'));
console.log(mod.upper('mango'));
console.log(mod.get_name());

Execute test.js to see the module in action:

$ node test.js
Packt
apple
MANGO
Packt

The assignment to module.exports approach is straightforward too. If you were to implement the previous module using the assignment method, this is how it would look like:

var secret = 'zoltan';

module.exports = {

  name: 'Packt',

  lower: function(input) {
    return input.toLowerCase();
  },

  upper: function(input) {
    return input.toUpperCase();
  },

  get_name: function() {
    return this.name;
  },

  get_secret: function() {
    return secret;
  }
  
};

There is an interesting thing about the second method of writing Node modules: you can assign any valid JavaScript object to the module.exports property, and it becomes the module. In the following example, we assign a function to the module.exports property:

module.exports = function(word) {
  
  var reversed = '';
  
  var i = word.length - 1;
  while (i> -1) {
    var letter = word[i];
    reversed += letter;
    i--;
  }
  
  return reversed;
};

Save the preceding code in a file named reverse.js. You can include it in the test.js file and use if for reversing text:

var reverse  = require('./reverse.js');
console.log(reverse('hippopotamus'));

Execute test.js again to see reverse.js in action:

$ node test.js
sumatopoppih

Using the assignment method, you can create Node modules to be of any valid JavaScript object type.

If you ever happen to have both the attachment and assignment methods defined in the same module file, the assignment method will take precedence.

It might sound a little strange, but every Express app is also a Node module. You might rarely use your web app like a regular Node module and include them in other apps, but there is something which will be an indispensable part of your app—its manifest file, package.json.

Node modules come with a manifest file named package.json, which contains details, such as its name, version, dependencies, and so on about the module.

Here is an example of an Express app's package.json file:

{
  "name": "application-name",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "start": "node app"
  },
  "dependencies": {
    "express": "3.2.6",
    "jade": "*",
    "stylus": "*"
  }
}

Among the various fields, dependencies is what would be of your prime interest. For an interactive guide to all the possible fields in a package.json file, visit http://package.json.nodejitsu.com/.

Any time you install a Node module in the application directory, the module will get added to the dependencies list with the version you specified. Of course, you can manually make new entries or update the version numbers of existing dependencies if you want to.

You may wonder what is the point of adding the modules in the dependencies when you already are installing them using npm. Well, if you start using a version control system such as Git or SVN, it doesn't make sense to include the installed Node modules in the repository. However it makes sense to include the package.json file, because with a simple npm install command in the app directory, you can reinstall the dependencies in one go.

It is advisable to use all other fields of the package.json file, but you certainly can't do without the dependencies key, if you are serious about your app.

By convention, the main file of the Express app is named app.js. You can rename it to anything you want, but it is generally not recommended to do so.

A middleware is a JavaScript function to handle HTTP requests to an Express app. It can manipulate the request and the response objects or perform an isolated action, or terminate the request flow by sending a response to the client, or pass on the control to the next middleware.

Middlewares are loaded in an Express app using the app.use() method.

Following is an example of a middleware. All it does is print the IP address of the client that made the request. Although it may seem like a trivial middleware, it gives you a very good overview of how middlewares work:

app.use(function(req, res, next) {
  console.log('Request from: ' + req.ip);
  next();
});

As you can see, a middleware is just a function that accepts three parameters: req, res, and next. The req parameter is the request object, the res parameter is the response object, and the next parameter is a reference to the next middleware in line. Any middleware can end a request by sending a response back to the client using one of the response methods on the res object. Any middleware that does not call a response method must call the next middleware in line, else the request will be left hanging in there.

Even though our middleware in the previous example was pretty simple, in most practical cases, middlewares will be created in a more complex fashion—they could be a JavaScript object defined right in the file, or might be included as a Node module.

This is how a middleware would look like if it were defined first and then passed to the app.use() method:

// define the middleware
var forbidder = function(forbidden_day) {

  var days = ['Sunday', 'Monday', 'Tueday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
  
  return function(req, res, next) {
    
    // get the current day
    var day = new Date().getDay();
    
    // check if the current day is the forbidden day
    if (days[day] === forbidden_day) {
      res.send('No visitors allowed on ' + forbidden_day + 's!');
    }
    // call the next middleware
    else {
      next();
    }
  }
};

// use the forbidder middleware
app.use(forbidder('Wednesday'));
// the router middleware goes here
app.use(app.router);

This middleware forbids visitors on your website on a certain day. Probably not a very useful middleware, but the intent is to show you how a middleware works.

One thing you might have noted is that we included the forbidder middleware before the router middleware. Does it make any difference? Oh yes, it does! A middleware included earlier takes precedence over those included later. So be careful about the order of inclusion.

If we were to rewrite the forbidder middleware as a Node module, we would need to first create the forbidder.js module file with the following content:

module.exports = function(forbidden_day) {

  var days = ['Sunday', 'Monday', 'Tueday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
  
  return function(req, res, next) {
    
    // get the current day
    var day = new Date().getDay();
    
    // check if the current day is the forbidden day
    if (days[day] === forbidden_day) {
      res.send('No visitors allowed on ' + forbidden_day + 's!');
    }
    // call the next middleware
    else {
      next();
    }
  }
};

Then, the module would be included in the app, and an instance of the module would be created:

var forbidder = require('./forbidder.js');

And the middleware would be added to the chain:

app.use(forbidder('Wednesday'));

The majority of top-level Express functionality is implemented via its built-in middlewares. An indispensable component of Express is the router middleware, which is responsible for routing the HTTP requests to your Express apps to the appropriate handler functions.

One might be tempted to think that when you make a request to your web app, there would be a corresponding JavaScript file that would be executed by Node. For example, to load the home page, there would be a file named home.js, for the contact page, contact.js, and so on.

That's not the case in an Express app. There is a single entry point for all the requests coming to the app—via app.js—which bootstraps the Express framework.

When an HTTP request arrives at your app, it goes through a stack of middlewares. All the middlewares in the chain have the capacity to modify the request and the response object in any form and manner, and that's how they work, as we learned in the last section.

Among the middlewares, which are include in Express, the most important is the router middleware, which gives Express the capability to define routes and handle them.

Here is a conceptualized representation of routes and their handlers:

The destinations of the HTTP request URIs are defined via routes in the app. Routes are how you tell your app "for this URI, execute this piece of JavaScript code". The corresponding JavaScript function for a route is called a route handler. It is the responsibility of the route handler to respond to an HTTP request, or pass it on to another handler function if it does not. Route handlers may be defined in the app.js file or loaded as a Node module.

Here is a working example of some routes and their handlers defined right in the app.js file:

var http = require('http');
var express = require('express');
var app = express();

app.get('/', function(req, res) {
  res.send('Welcome!');
});

app.get('/hello.text', function(req, res) {
  res.send('Hola!');
});

app.get('/contact', function(req, res) {
  res.render('contact');
});

http.createServer(app).listen(3000, function(){
  console.log('Express server listening on port ' + 3000);
});

Defining the routes and their handlers in the app.js file may work fine if the number of routes is relatively few. It becomes messy if the number of routes starts growing. That's where defining the routes and their handlers in a Node module comes in handy. If we were to modularize the routes we defined earlier, here is how it would look like.

The following is the content of the routes.js Node module:

module.exports = function(app) {
  
  app.get('/', function(req, res) {
    // Send a plain text response
    res.send('Welcome!');
  });
  
  app.get('/hello.text', function(req, res) {
    // Send a plain text response
    res.send('Hola!');
  });
  
  app.get('/contact', function(req, res) {
    // Render a view
    res.render('contact');
  });
};

The modified app.js file would look like the following now:

var http = require('http');
var express = require('express');
var app = express();
var routes = require('./routes')(app);

http.createServer(app).listen(3000, function(){
  console.log('Express server listening on port ' + 3000);
});

A request handler can send a response back to the client using one of the response methods in the response object. The act of sending a response effectively terminates the request flow to any other route handler.

Views are special files in an Express app, which are sent as an HTML response after Express processes them. Express views support multiple layout and CSS preprocessor engines. In this book, we will focus on Jade for HTML and Stylus for CSS.

Express is built on top of Node's HTTP/HTTPS API. When one hears something like that, often it so happens that the underlying API is insulated by the framework, but it is not the case in Express. The Node HTTP/HTTPS API is very much accessible from the Express framework—the req and res objects are extensions of the req and res socket objects in a plain vanilla Node HTTP server.

So, anytime you feel the need to hack a little deeper, you can go ahead and work on the original Node objects and their properties and methods.

While we are at it, I would like to stress the point that not only is the HTTP/HTTPS API available for Express, but the whole of Node API is available from Express. Reading up the complete Node documentation will help you become a more efficient Express developer—you will understand the underlying mechanism better, write better middleware and modules for your apps, and have more control over the framework.