Building a Scalable Load Balancer with Node.js and Express

Load balancing is a critical component of any scalable web application. It helps distribute incoming client requests evenly across multiple backend servers, optimizing resource utilization and ensuring high availability.

In this comprehensive tutorial, we will build a highly flexible load balancer using Node.js and Express that can handle multiple heterogeneous backend servers.

Overview

Here are the core features we will implement in our load balancer:

• Accept HTTPS requests and terminate SSL
• Load balance across multiple backend application servers
• Health check endpoints on each server for monitoring status
• Weighted round robin algorithm to distribute loads based on server capacity
• Session affinity/stickiness — route requests from same client to same backend
• Graceful handling of servers being added or removed
• Customizable with different balancing strategies
• Detailed logging and metrics for monitoring

We will use a modular design pattern to make it easy to modify or extend components later.

Our load balancer will run as a standalone Node.js application. We will use Express for handling overall application routing and logic. And we’ll create separate router modules for handling health checks, metrics, etc.

The backend application servers are simple Express apps running on different ports that respond with their hostname, to simulate a real multi-server environment.

Project Setup

Let’s start by initializing a new Node.js project.mkdir load-balancer
cd load-balancer
npm init -y

Now we’ll install Express and some other dependencies:npm install express http-proxy-middleware cluster cookie-parser

• express — web framework
• http-proxy-middleware — module for proxying requests
• cluster — core Node module for multi-threaded load balancing
• cookie-parser — parse cookies from requests

SSL Termination

Our load balancer needs to accept HTTPS traffic and terminate SSL before passing requests to the backend servers.

Install the HTTPS module:npm install https

In index.js let’s add the HTTPS server:

const fs = require(‘fs’); const https = require(‘https’); const options = { key: fs.readFileSync(‘./ssl/key.pem’), cert: fs.readFileSync(‘./ssl/cert.pem’), }; https.createServer(options, app).listen(443, () => { console.log(‘Load balancer started on port 443’); });

We need to generate a self-signed certificate and key pair for testing. Let’s create an ssl folder and use OpenSSL:mkdir ssl
cd ssl
openssl req -nodes -new -x509 -keyout key.pem -out cert.pem

This will generate key.pem and cert.pem files that our HTTPS server can use.

Now our load balancer will run as an HTTPS server.

Load Balancing Logic

The core load balancing logic will be implemented in routes/proxy.js which will forward requests to backend servers.

Install the http-proxy-middleware package:npm install http-proxy-middleware

Now create routes/proxy.js:

const express = require('express');
const proxy = require('http-proxy-middleware');

const router = express.Router();

const servers = [
{
host: 'localhost',
port: 3000,
weight: 1,
},
// Add more servers here
];

// Proxy middleware configuration
const proxyOptions = {
target: '',
changeOrigin: true,
onProxyReq: (proxyReq, req) => {
// Add custom header to request
proxyReq.setHeader('X-Special-Proxy-Header', 'foobar');
},
logLevel: 'debug'
};

// Next server index
let currIndex = 0;

// Get next server
function getServer() {

// Round robin
currIndex = (currIndex + 1) % servers.length;

return servers[currIndex];
}

// Proxy requests
router.all('*', (req, res) => {

// Get next target server
const target = getServer();
proxyOptions.target = `http://${target.host}:${target.port}`;

// Forward request
proxy(proxyOptions)(req, res);
});

module.exports = router;

This implements a basic round-robin proxy routing.

In index.js we can mount it:const proxyRouter = require(‘./routes/proxy’);

app.use(‘/app’, proxyRouter);

Now requests to /app will be proxied to the backends.

Let’s also add a simple HTTP backend server that responds with its hostname:

server.js:

const express = require('express');

const app = express();

app.get('/app', (req, res) => {
res.send(`Hello from server! Host: ${process.env.HOSTNAME}`);
});

app.listen(3000, () => {
console.log('Backend server running on port 3000');
});

We can test the proxying and see the request getting forwarded:$ curl -k https://localhost/app
Hello from server! Host: host1

Next we’ll implement other important load balancing features.

Health Checks

Health checks make sure we only send traffic to “healthy” backend servers.

Let’s create a health check router in routes/health.js:

const express = require('express');
const axios = require('axios');

const router = express.Router();

router.get('/health', async (req, res) => {

  const results = [];

  // Loop through servers and health check each 
  for (let i = 0; i < servers.length; i++) {
    const server = servers[i];

    try {
      const response = await axios.get(`http://${server.host}:${server.port}/app/healthcheck`);
      // Check status
      if (response.status === 200) {
        results.push({
          id: server.id,
          status: 'passing'
        });  
      } else {
        results.push({
          id: server.id,
          status: 'failing' 
        });
      }
      
    } catch (error) {
      results.push({
        id: server.id,
        status: 'failing'
      });
    }
  }

  // Return summarized results
  res.json(results); 
});

module.exports = router;

This guarantees we won’t proxy to any backends that are unhealthy.

Weighted Round Robin

Simple round robin balancing treats all servers as equal. But we may want to distribute more load to servers with higher capacity.

Let’s implement weighted round robin:

// Server object
{
  host: 'localhost',
  port: 3000,
  // Higher weighted servers get more requests
  weight: 1 
}

// Total weights
let totals = [];

// Generate list of cumulative weights 
function initWeights() {

  totals = [];
  let runningTotal = 0;

  for (let i = 0; i < servers.length; i++) {
    runningTotal += servers[i].weight;
    totals.push(runningTotal); 
  }

}

function getServer() {

  const random = Math.floor(Math.random() * totals[totals.length - 1]) + 1;

  // Find server at index for this weight
  for (let i = 0; i < totals.length; i++) {
    if (random <= totals[i]) {
      return servers[i];
    }
  }

}

This assigns “shares” of requests based on server weight.

Now we can configure servers with different weights to distribute loads accordingly.

Session Affinity

With session affinity, requests from the same client session always get routed to the same backend server. This keeps their user session state consistent.

We can use cookie-based affinity:

// Generate cookie name
const COOKIE_NAME = 'lb-affinity';

app.use(cookieParser());

router.all('*', (req, res) => {

  // No affinity for first request
  if (!req.cookies[COOKIE_NAME]) { 
    // Set cookie 
    res.cookie(COOKIE_NAME, selectedServer.id, {
      httpOnly: true
    });

  } else {
    // Re-route request to previously selected server
    const affinityId = req.cookies[COOKIE_NAME];
    
    selectedServer = servers.find(s => s.id === affinityId);
  }

  // Route request  
});

Now subsequent requests from the user will hit the same backend server.

This sticks a client to one server for their session.

Graceful Server Updates

We don’t want to abruptly terminate live client connections if a server is added/removed from the pool.

To gracefully handle updates:

const draining = [];
const drained = [];

// Drain server - leave in proxying but don't add new clients
function drain(serverId) {
  draining.push(serverId);
} 

// Mark server as fully drained 
function drained(serverId) {
  drained.push(serverId);
}

router.all('*', (req, res) => {

  if (!req.cookies[COOKIE_NAME]) {

    const server = // Select server
    
    // Check if draining
    if (draining.includes(server.id)) {
      return sendToBackup(req, res); // Bypass if draining
    }

  } else { 

    // Route to selected server

  }

})

// Fully remove from pool
function removeServer(serverId) {

  const index = servers.findIndex(s => s.id === serverId);

  if (index !== -1) {
    servers.splice(index, 1);
  }

  drained.splice(drained.indexOf(serverId), 1);

}

// Add back to pool 
function addServer(server) {
  servers.push(server);
}

This ensures graceful draining of connections from servers being updated.

Customizable Balancing Strategies

For flexibility, we can setup our load balancer to support different balancing strategies that can be swapped in and out.

First, define a strategy interface:

// Strategy interface 
function Strategy() {
  this.select = null;
}

// Random strategy
function RandomStrategy() {

  // Set select method
  this.select = function() {
    // Return random server
  }

}

// Round robin strategy
function RoundRobinStrategy() {
  
  // Set select method
  this.select = function() {
    // Return next server in round robin fashion
  }

}

// Map strategies to readable names
const Strategies = {
  'random': RandomStrategy,
  'roundrobin': RoundRobinStrategy  
};

This makes it easy to implement new strategies and swap them in.

Metrics and Logging

Capturing metrics and logs is vital for monitoring the load balancer.

We can utilize the onProxyReq and onProxyRes callbacks:

// Proxy options
const options = {
  onProxyReq: (proxyReq, req) => {
    // Log details like request timestamps, headers etc
    
  },
  onProxyRes: (proxyRes) => {
    // Log response status, time taken, etc.

  }
}

And log key metrics like:

• Requests per second
• Request latencies
• Traffic distribution across backends
• Error rates
• Health check latencies

These metrics can be shipped to a stats collector like StatsD.

Detailed logs allow tracing requests and debugging issues.

Conclusion

In this article we implemented a full-featured load balancer in Node.js with:

• SSL termination
• Health checks
• Flexible server selection with customizable strategies
• Graceful handling of topology changes
• Detailed metrics and logs

The main benefits are:

High Availability — failed servers are quickly taken out of rotation

Flexibility — support heterogeneous backends, modify balancing strategies

Scalability — scale across multiple servers, geo regions

Visibility — metrics provide insights into performance and utilization

There are many potential enhancements like caching, access control, and more.

learn more about
NestJS vs Go: Performance comparison for JWT verify and MySQL query