Consider a scenario, in the node server, we receive a request and have tons of computation before sending the response. This computation task is in javascript and is not delegated to the OS or thread pools. During this computation, the event loop can not take the next request.
const doSomeWork = duration => {
const start = Date.now();
while (Date.now() - start < duration) {}
};
app.get('/', (req, res) => {
doSomeWork(5000);
res.send('Hi there');
});
app.listen(8000);
To overcome the issue, we can introduce clustering and improve the performance.
Clustering
Since the event loop is single-threaded, any time-consuming operation will pause the loop for further I/O for a certain period of time. There are two ways, we can somehow mitigate the performance impact,
- Cluster mode
- Worker threads
With clustering, there is a parent process, called cluster manager.
Cluster manager,
- Monitor the health of the instances
- Can start/stop/restart the instances
- Can send data to the instances
Cluster manager does not Handle requests or Fetch data from DB.
Without clustering, when we run a node.js app, it went through the index.js and creates an instance to handle the incoming request.
With clustering, initially, an instance called Cluster is created. Then the Cluster Instance is responsible for creating a child instance, using child.fork(). These child instances are responsible for handling incoming requests.
const cluster = require('cluster');
if (cluster.isMaster) {
cluster.fork();
cluster.fork();
} else {
const doSomeWork = duration => {
const start = Date.now();
while (Date.now() - start < duration) {}
};
app.get('/', (req, res) => {
doSomeWork(5000);
res.send('Hi there');
});
app.listen(8000);
}
Advantage of Clustering
Let's consider the express server route,
const doSomeWork = duration => {
const start = Date.now();
while (Date.now() - start < duration) {}
};
app.get('/fast', (req, res) => {
res.send('Fast route');
});
app.get('/slow', (req, res) => {
doSomeWork();
res.send('Fast route');
});
With this code, if we call, /slow and almost immediately /fast, the following scenario happens,
- We first hit the
/slow and the event loop will not process any further requests in the next 5 seconds.
- After
/slow is finished, the /fast route comes to play and gets executed.
It appears, that even the /fast is super fast but due to the event loop in a single thread and the previous route is taking 5 seconds, it can not be executed immediately.
In this case, we can make use of the clustering as follows,
if (cluster.isMaster) {
cluster.fork();
cluster.fork();
} else {
const doSomeWork = duration => {
const start = Date.now();
while (Date.now() - start < duration) {}
};
app.get('/fast', (req, res) => {
res.send('Fast route');
});
app.get('/slow', (req, res) => {
doSomeWork();
res.send('Fast route');
});
}
Here, with clustering, we make two of the node.js event loop instances. In this case, when we call /slow and almost immediately /fast,
- First event loop instance will take care of
/slow
- The second event loop instance will take care of
/fast while the /slow is still being proceeded by the first event loop
So this is a practical way, where clustering can play a huge benefit inside the app. When there is some route in-app, that takes longer to process and some other route that is fast, using clustering, we can spin up multiple servers and can achieve a faster response time.
Limitation of Clustering
There are a couple of corner cases in clustering to which we should be concerned.
Before we get into the example let's make some assumption
- The machine, running the code, has two cores (Duel core machine)
- The hashing algorithm requires 1 second to be executed in one single thread
- Our thread pools have only one thread at a single time
Now, consider the following code,
process.env.UV_THREADPOOL_SIZE = 1;
if (cluster.isMaster) {
cluster.fork();
} else {
app.get('/', (req, res) => {
runHash(() => {
return res;
});
});
}
In this case, when we send 1 request, it will be executed in 1 second and the response time will be the same.
Consider making 2 requests concurrently. Since there is only one thread, the first hash will take 1 second and then the 2nd hashing starts working. So the second request will be completed after 2 seconds.
How about, we make use of clustering by making 2 children here,
process.env.UV_THREADPOOL_SIZE = 1;
if (cluster.isMaster) {
cluster.fork();
cluster.fork();
} else {
app.get('/', (req, res) => {
runHash(() => {
return res;
});
});
}
Now when we send 2 requests, concurrently, both two child servers take one and execute in 1 second in total. This is the place, clustering comes to play.
Since using the 2 children server can optimize operation, can we add more and more children and get more optimized results? How about making 6 children?
process.env.UV_THREADPOOL_SIZE = 1;
if (cluster.isMaster) {
cluster.fork();
cluster.fork();
cluster.fork();
cluster.fork();
cluster.fork();
cluster.fork();
} else {
app.get('/', (req, res) => {
runHash(() => {
return res;
});
});
}
Now we have 6 children and we will send 6 requests concurrently. Surprisingly, we can notice, that all the hashing will be completed in around 3.5 ~ 4 seconds.
This is because we assume our machine has only 2 threads and can not process more at a single time. So all 6 hashing try to be resolved simultaneously and the limited 2 cores do partial work on all. This is why it even takes more time.
So to get an optimized response, it is useful to adjust child server numbers according to the number of cores in the server.
Handle clustering in your application
Instead of manually running clustering, production can better handle using pm2.
PM2 is a cluster management tool that can handle clustering very efficiently.
- Can spin up child server according to the logical core of the server
- Can spin up child server by specifying the number
- Can inspect, monitor, and log the running instances
- Can start/end/restart the child instances
Difference Between Clustering and worker threads.
References: Node JS: Advanced Concepts By Stephen Grider
