Excerpt: PM2 sees a dead process, not a stuck event loop. This is an external healthcheck on bash and a systemd timer that calls pm2 restart on its own, with a cooldown against restart loops and an optional Telegram notification.
PM2 restarts a process when it dies. It has no real healthcheck: nothing that probes the app from outside and hits restart on its own when the app stops responding. If you already went through the diagnostic steps in the memory leak article and confirmed your process isn't dying, just going quiet, the fix isn't another monitoring tool. It's a script that restarts the hung process while you're asleep.
There are three different reasons an app can go quiet, and conflating them causes real bugs. The process can crash - PM2 handles that on its own. The event loop itself can get blocked - heavy synchronous code, a loop that never awaits, a large `fs.*Sync` call - and then nothing on that process answers, not even an empty `/health` route. Or a specific request can hang - a database call with no timeout on a connection that dropped silently - while the event loop itself stays free and keeps serving everything else. PM2 catches neither of the last two, and `max_memory_restart` doesn't help unless the failure happens to be memory growth. The fix is an external healthcheck: a bash script that hits the app over HTTP on 127.0.0.1, tracks consecutive failures in a state file, and calls `pm2 restart` once it hits a threshold (two failures in a row by default). Scheduling runs through a systemd timer that logs every run to journalctl. A Telegram notification is an optional layer on top, not a requirement for the healthcheck to work. Restarting genuinely fixes a hung process, but it does not fix a database that's down somewhere else - if the probe depends on an external service rather than just the app itself, a restart can end up hammering a perfectly healthy process while the real problem sits untouched. Below covers how to dodge that trap, plus a circuit breaker in the script that stops trying once restarts clearly aren't helping.
PM2 watches whether the operating system process is alive: whether it exists, whether it exited with an error, whether the OOM killer took it out. The status `online` means exactly that and nothing more. It says nothing about whether the process is answering requests.
Worth separating two failures that look identical from the outside - the site is down - but need different fixes.
The event loop itself is blocked. The event loop is the mechanism Node.js uses to work through incoming tasks one at a time: HTTP requests, timers, callbacks coming back from network calls. Only synchronous, blocking code can jam it: a heavy loop with no await inside, `fs.readFileSync` on a large file, `crypto.pbkdf2Sync` with a lot of iterations, an accidental infinite `while` with nothing async in it. While that code runs, the process can't handle anything else, including the emptiest possible `/health` route on the same server - because serving even that still needs the same event loop.
The event loop is fine, but one specific request is hung. A normal database or external API call in Node.js is async: while it waits, the event loop is free and keeps serving everything else. If the network client has no timeout set and the connection drops silently (not a clean TCP reset, just gone, say behind a firewall or a dead link), that promise may never resolve - but that only blocks that one request and whatever depends on it. The rest of the process, including an empty `/health` route, answers normally. This is the case PM2 stays blind to the longest: the process is alive, CPU isn't climbing, memory isn't climbing - some requests just hang forever.
`max_memory_restart` in your PM2 config covers exactly one failure mode: memory usage climbing past a threshold, the scenario from the leak article. It catches neither a blocked event loop nor a hung request - memory can sit perfectly still in both cases.
Symptom | Does PM2's own restart fire? | Does the healthcheck fire? |
|---|---|---|
Unhandled exception kills the process | Yes | Not needed, but fires too |
Event loop blocked by synchronous code, process alive | No | Yes - nothing answers, not even an empty /health |
A specific request hangs with no timeout (e.g. a DB call), event loop free | No | Yes, if the probed route shares the same connection/pool as the hung request |
Memory leak pushes RSS past `max_memory_restart` | Yes | Yes, redundant coverage |
Slow leak stays under the threshold, but GC pauses tank response times | No | Yes, if pauses blow past the request timeout |
nginx or Cloudflare Tunnel goes down, process itself is fine | No | No, and that's the correct behavior - see the 127.0.0.1 section below |
An external database or API is down, process and event loop both fine | No | Depends what you're probing - see liveness vs readiness below |
The tempting shortcut is a bare `/api/health` route that just returns `200 OK` and touches nothing else. The problem: that only catches a blocked event loop, not a hung request to a specific dependency - if the memory-leak scenario from the first article in this series is exactly that (a hang on a specific database call), an empty health route answers instantly, because it never goes near that code path, while the real homepage sits there not answering anyone.
So should you always probe something that touches the same database or cache as real traffic? Not always - and there's a real trap here. If that route depends on an external database that goes down entirely for, say, 20 minutes, the process and event loop are both completely healthy, and the healthcheck honestly sees failure after failure and keeps restarting Node.js. Restarting the process does not fix a downed database: the app comes back up, immediately hits the same dead database, fails the probe again - and you get exactly the restart loop you were trying to avoid.
The distinction is between liveness (is the process itself alive and able to do work at all) and readiness (can it fully serve a user right now). For automatic restart, a liveness probe is safer - one that reacts to a condition a restart can actually fix. A probe tied to an external dependency is useful for monitoring and alerting, but shouldn't decide restarts on its own, unless you've specifically proven that it's an internal connection pool locking up (not the external service itself), and that restarting actually clears it.
A reasonable middle ground for a typical small VPS app: probe a route that does something real, not a stub, but watch separately for a spike specifically in failures tied to one external dependency - that's exactly where the circuit breaker in the script below (stop after N restarts, alert instead) earns its keep rather than being a nice-to-have.
A dashboard, a Slack alert, a monitoring email all do the same thing: tell a human that something is wrong. That human then has to wake up at 3am, open a laptop, SSH in, and run `pm2 restart` by hand. The app stays down the whole time they're doing that.
The healthcheck in this article runs that same restart on its own, within a minute or two of the app going quiet, and it still leaves a trail: a line in journalctl, and a Telegram message if you've wired one up. There's still something to review in the morning. You just get to review it after coffee instead of losing sleep over it.
The logic is deliberately simple, and that's the point - there's almost nothing in it that can break.
The threshold is two failures, not one, to avoid restarting a perfectly healthy process over a brief network blip or a garbage collection pause. One failure is noise. Two in a row, a minute apart, is a signal.
There's a separate question worth getting right: what exactly to probe. A tempting shortcut is a bare `/api/health` route that just returns `200 OK` and touches nothing else. That's a trap. If the app hangs on a specific database call (the scenario from the memory leak article), an empty health route answers instantly, because it never goes near that code path, while the actual homepage sits there not answering anyone. Probe whatever a real user request would hit. In most cases that's just the homepage. A dedicated `/health` route only earns its keep if it exercises the same database connection or cache the real traffic depends on, not a code path that exists in isolation from it.
Save the script under the same user PM2 runs as (`deploy` in this series - swap in your own). Adjust the port, path, and app name if yours differ from the defaults below.
#!/usr/bin/env bash
set -euo pipefail
# what we're checking and how
APP_NAME="web" # pm2 process name, see `pm2 list`
URL="http://127.0.0.1:3000/" # hit the local port, not the public domain
TIMEOUT=5 # seconds before a check counts as failed
FAIL_THRESHOLD=2 # consecutive failures before a restart
COOLDOWN=300 # don't restart more than once per 5 minutes
# guard against an endless restart loop when restarting doesn't actually help
MAX_RESTARTS=3 # no more than this many restarts...
RESTART_WINDOW=1800 # ...within this many seconds (30 minutes)
STATE_DIR="/var/lib/healthcheck"
FAIL_FILE="$STATE_DIR/failcount"
LAST_RESTART_FILE="$STATE_DIR/last-restart"
RESTART_LOG="$STATE_DIR/restart-log" # one timestamp per line, one per restart
mkdir -p "$STATE_DIR"
[ -f "$FAIL_FILE" ] || echo 0 > "$FAIL_FILE"
[ -f "$RESTART_LOG" ] || : > "$RESTART_LOG"
# stop two runs of the script (say, a manual test landing on a timer tick) from
# racing on the same state files
exec 9>"$STATE_DIR/lock"
flock -n 9 || { echo "$(date -Is) SKIP: another run is already in progress"; exit 0; }
PM2_BIN="${PM2_BIN:-pm2}" # if systemd can't find pm2 on PATH, hardcode the output of `which pm2` here
notify() {
# $1 - message text; silently does nothing if TELEGRAM_* isn't set
if [ -n "${TELEGRAM_BOT_TOKEN:-}" ] && [ -n "${TELEGRAM_CHAT_ID:-}" ]; then
curl -fsS --max-time 5 -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
-d chat_id="${TELEGRAM_CHAT_ID}" \
-d text="$1" \
> /dev/null || echo "$(date -Is) WARN: Telegram notification failed to send"
fi
}
if curl -fsS --max-time "$TIMEOUT" "$URL" -o /dev/null; then
echo 0 > "$FAIL_FILE"
echo "$(date -Is) OK $URL"
exit 0
fi
fails=$(cat "$FAIL_FILE")
fails=$((fails + 1))
echo "$fails" > "$FAIL_FILE"
echo "$(date -Is) FAIL $URL (streak: $fails)"
if [ "$fails" -lt "$FAIL_THRESHOLD" ]; then
exit 0
fi
now=$(date +%s)
last=$(cat "$LAST_RESTART_FILE" 2>/dev/null || echo 0)
if [ $((now - last)) -lt "$COOLDOWN" ]; then
echo "$(date -Is) SKIP: $APP_NAME was already restarted recently, waiting out the ${COOLDOWN}s cooldown"
exit 0
fi
# how many restarts happened within the last RESTART_WINDOW seconds - also
# trims the log down to recent entries so it doesn't grow forever
cutoff=$((now - RESTART_WINDOW))
awk -v c="$cutoff" '$1 > c' "$RESTART_LOG" > "$RESTART_LOG.tmp" && mv "$RESTART_LOG.tmp" "$RESTART_LOG"
recent=$(wc -l < "$RESTART_LOG")
if [ "$recent" -ge "$MAX_RESTARTS" ]; then
echo "$(date -Is) STOP: $APP_NAME has been restarted $recent times in the last $((RESTART_WINDOW / 60)) minutes - restarting isn't helping, backing off"
notify "healthcheck: ${APP_NAME} on $(hostname) keeps failing right after every restart (${recent} attempts in $((RESTART_WINDOW / 60)) min). Auto-restart stopped, needs a human."
exit 0
fi
echo "$(date -Is) RESTART $APP_NAME after $fails consecutive failures (restarts in window: $recent)"
"$PM2_BIN" restart "$APP_NAME"
echo "$now" > "$LAST_RESTART_FILE"
echo "$now" >> "$RESTART_LOG"
echo 0 > "$FAIL_FILE"
notify "healthcheck: restarted ${APP_NAME} on $(hostname) after ${fails} failed checks"
The state directory has to exist before the first run. A regular user can write into a directory they already own, but can't create a new one under `/var/lib` on their own:
sudo install -d -o deploy -g deploy /var/lib/healthcheck
sudo install -m 755 healthcheck.sh /usr/local/bin/healthcheck.sh
Test it by hand before trusting the schedule. A healthy run should print a line like 2026-09-14T03:12:01+00:00 OK http://127.0.0.1:3000/:
sudo -u deploy /usr/local/bin/healthcheck.sh
To exercise the failure path, first reset the state files - if the script has run before, a leftover fail count or a recent restart timestamp can let the cooldown silently swallow your test:
sudo -u deploy sh -c 'echo 0 > /var/lib/healthcheck/failcount'
sudo rm -f /var/lib/healthcheck/last-restart
Then stop the process and run the script twice by hand. The second run should print RESTART web after 2 consecutive failures, and `pm2 list` should show a freshly reset uptime for `web` right after:
pm2 stop web
sudo -u deploy /usr/local/bin/healthcheck.sh
sudo -u deploy /usr/local/bin/healthcheck.sh
pm2 list
The path from a reader to your app, per the Next.js VPS deploy article, runs through Cloudflare Tunnel, then nginx, then finally into the process on localhost. Point the healthcheck at the public domain and you're testing the whole chain: DNS, the tunnel, nginx, TLS. Any hiccup anywhere in that chain (the tunnel daemon restarting, an nginx reload, a short DNS blip) reads exactly like a dead process, and the script restarts a perfectly healthy app for a problem that has nothing to do with it.
Hitting `127.0.0.1:3000` talks directly to the thing PM2 actually supervises, the one component this script is allowed to fix by restarting. If the tunnel or nginx is broken, that's a separate problem, and restarting the app process won't touch it.
`systemd` is the standard service manager on Linux; besides keeping long-running processes alive, it can also run one-off tasks on a schedule through a pair of units: a `.service` describing the task, a `.timer` describing when to run it. Unlike cron, every run gets logged to journalctl along with the script's full output, instead of vanishing into root's mail, which on a fresh VPS usually isn't even set up.
The service unit is a `oneshot`, a task that runs once and exits, unlike a long-running process such as the app itself:
[Unit]
Description=Probes 127.0.0.1:3000 and restarts web via pm2 on consecutive failures
After=network.target
[Service]
Type=oneshot
User=deploy
WorkingDirectory=/home/deploy
Environment=PATH=/home/deploy/.nvm/versions/node/v20.19.0/bin:/usr/local/bin:/usr/bin:/bin
EnvironmentFile=-/etc/healthcheck.env
ExecStart=/usr/local/bin/healthcheck.sh
The timer fires that service once a minute:
[Unit]
Description=Runs healthcheck.service every minute
[Timer]
OnBootSec=1min
OnUnitInactiveSec=1min
AccuracySec=5s
Unit=healthcheck.service
[Install]
WantedBy=timers.target
Enable it and check:
sudo systemctl daemon-reload
sudo systemctl enable --now healthcheck.timer
systemctl list-timers | grep healthcheck
journalctl -u healthcheck.service -f
`list-timers` should show a `NEXT` column pointing to a run within the next minute. Watching `journalctl -f` for a couple of minutes should produce `OK http://127.0.0.1:3000/` lines, confirming the timer is actually firing and not just sitting enabled.
If you'd rather use cron, that works too, but it has the same environment problem systemd does, and you have to fix it in two places at once: the `PATH` itself, and where the log is even allowed to go. A regular user like `deploy` can't write to `/var/log` - the command fails on the redirect before the script ever runs. And `PM2_BIN` alone isn't enough either: if PM2 was installed via nvm, its shebang is `#!/usr/bin/env node`, and if the nvm directory holding `node` isn't in `PATH`, the full path to `pm2` still won't help since the process has nothing to run itself with. Add it with `crontab -e`, run as `deploy`:
PATH=/home/deploy/.nvm/versions/node/v20.19.0/bin:/usr/local/bin:/usr/bin:/bin
PM2_BIN=/home/deploy/.nvm/versions/node/v20.19.0/bin/pm2
* * * * * /usr/local/bin/healthcheck.sh >> /home/deploy/healthcheck.log 2>&1
Swap in your actual Node version in both paths (`ls ~/.nvm/versions/node/` shows what's installed). The log now writes to `deploy`'s home directory, which it can actually write to, not `/var/log`. Beyond that, the main tradeoff versus the systemd timer is how you read the logs afterward: `journalctl -u healthcheck.service` filters by time and level out of the box, while a flat log file needs `tail`/`grep` by hand - if that's not a dealbreaker, the systemd timer is still the more predictable choice.
The whole loop, start to finish: a wedged app that PM2 still honestly calls "online", a manual failing probe, the fail counter climbing, an automatic restart triggered by the timer with nobody touching a keyboard, and the journalctl entry proving it:
The healthcheck works without this step: it restarts the process regardless of whether notifications are set up. A Telegram message just saves you from opening journalctl every morning to check whether anything happened overnight.
Get a bot token from `@BotFather` on Telegram, covered in the Telegram bot on a VPS article: message `/newbot`, get back a string like `123456789:AA...`. The `chat_id` is where the message goes. The cleanest way to find it, no third-party bots involved: send your freshly created bot any message, then open (or `curl`) https://api.telegram.org/bot<token>/getUpdates - the JSON response includes your message, and in it, message.chat.id is the number you need.
Keep the values in a separate env file rather than the script itself, so the token doesn't travel with every copy of the script:
TELEGRAM_BOT_TOKEN=123456789:AA...
TELEGRAM_CHAT_ID=987654321
sudo install -m 600 -o deploy -g deploy healthcheck.env /etc/healthcheck.env
Mode `600` and owner `deploy`, the same reasoning as the bot token in the earlier article: only the user the service runs as can read the file, nobody else.
If the app hangs once a day for some random reason, the healthcheck just fixes it once a day and that's the end of it. Worse is a non-random cause: a broken release that wedges the process a minute after every start, or an external database that's down for a while. Without a guardrail, the script would happily restart the process forever, producing the appearance of a working system instead of an actual signal that something's broken - and if the cause isn't even in the app, a restart doesn't fix anything at all.
Worth being precise about what protects against what. `COOLDOWN=300` in the script does not cap the total number of restarts - it only stops `pm2 restart` from firing more than once every five minutes. If the underlying problem doesn't go away, a script with only a cooldown will dutifully restart a broken process every five minutes forever. That's not a guard against a restart loop, it's just a slower restart loop.
The real guard is the `MAX_RESTARTS`/`RESTART_WINDOW` counter in the script above: no more than three restarts in the last 30 minutes, then the script stops touching the process and only sends a notification. That's a circuit breaker in the literal sense - it breaks the "failed -> restarted -> failed again" chain once restarting clearly isn't the fix, and leaves the decision to a human instead of hammering away blindly.
Separate from that is PM2's own config, covered in the Next.js VPS deploy article: `max_restarts` and `exp_backoff_restart_delay` protect against a different scenario - PM2 itself trying, on its own, to bring back a process that keeps exiting right after it starts. That's PM2 guarding against its own crash-restart attempts; it has nothing to do with restarts the external healthcheck script triggers, and it won't stop the loop of "healthcheck restarts it -> PM2 starts it -> app hangs again without crashing -> healthcheck restarts it again." That's exactly why the circuit breaker needs to live in the healthcheck script itself.
PM2 only restarts a process when it actually exits; a hung-but-alive process is left alone indefinitely. You need an external healthcheck: a script that probes the app over HTTP from the outside, tracks consecutive failures, and calls `pm2 restart` once it hits a threshold. The full script and its systemd timer are above.
It's a small standalone bash script that curls the app's local address every minute or two, keeps a consecutive-failure count in a file on disk, and runs `pm2 restart ` once that count hits a threshold. Add a cooldown so it doesn't hammer restart on every single check if the underlying problem isn't fixed.
PM2 tracks whether the operating system process is alive, not whether it's answering requests. Usually it's one of two things: either the event loop itself is blocked by synchronous code and answers nothing at all, or one specific request hung with no timeout (a database call, say) while the event loop stays free and keeps serving everything else. Either way the process is still running, so PM2 keeps reporting it as healthy.
A timer logs every run and its output to journalctl, so you can see the full history with `journalctl -u healthcheck.service`, while cron by default mails root, an address that's usually not even configured on a fresh VPS. A timer also waits for the previous run to finish and won't silently drop a run that produced no output.
Create a bot with `@BotFather`, grab the token and your chat ID, and add a `curl -X POST` to `https://api.telegram.org/bot/sendMessage` right after the `pm2 restart` line in the script. It's entirely optional: the healthcheck restarts the process with or without it, the notification just saves you from checking the logs by hand every morning.
This healthcheck fixes one specific gap: a process that's hung but still alive. It says nothing about shipping new versions without downtime, and it doesn't lock down the server itself against traffic on ports you never meant to open. Both are next in this VPS series: zero-downtime deploys, and hardening nginx and the firewall with ufw. If you haven't read the earlier pieces, start with deploying Next.js on a VPS and the general guide to deploying a service on a server.