Your Server Went Offline. Who Sends the Alert Now?

The checker needs somewhere else to stand

Think of two roles. Target is the website you want to watch. Observer is the machine that makes the request, evaluates the response and sends the notification.

If both roles share one VPS, a single machine failure can remove both. Another process, container or user account on that same VPS does not solve this particular problem.

Watch the public path your readers use

Use a separate Linux machine that can stay on and run the monitor. A laptop can help with a short experiment, but its sleep, shutdown and network changes make it a poor unattended observer.

Different infrastructure can reduce shared failures, but this lesson does not require buying a particular service. Use an existing suitable machine or an external monitoring service with equivalent checks.

Use the real public HTTPS hostname. Do not replace it with localhost, a private address or a local hosts-file override. Those can bypass parts of the route visitors depend on.

If a CDN serves your site, checking the public hostname includes that layer. A cached article may remain available while the origin fails. That is a useful result for that article, but it says nothing about an uncached login or checkout.

Move the check, then verify it again

On the separate Linux machine, follow the private setup in Lesson 02. Save the same monitor.py program and create a private settings.json for the public page and expected HTML marker.

Enter the bot token locally. You do not need to expose your website’s files, database or administrator credentials to the observer. It requests a public page and sends its result.

python3 "$HOME/.local/share/sa-website-monitor/monitor.py" --test-alert
python3 "$HOME/.local/share/sa-website-monitor/monitor.py"

Confirm the test message arrives, then read the initial page result. This observer has its own saved state, so its first normal run can send an initial notification even if the old machine already reported UP.

Before leaving it unattended, pause the old job if you are replacing it. Comment out only the old monitor’s crontab line. Keep the external observer as the active checker.

Make a failure without breaking your website

We will give the observer a deliberately invalid destination, confirm that the request fails, and send a clearly labelled test notification. Then we will check your real page.

The test uses .invalid, a reserved domain ending intended for invalid names. It does not require taking your server offline, changing DNS or editing your real settings.

Create this file on the observer:

nano "$HOME/.local/share/sa-website-monitor/test-external.py"

Paste the program below. It imports the check and notification functions from Lesson 02; it does not run the monitor’s normal startup routine.

#!/usr/bin/env python3
import importlib.util
import json
import os
from pathlib import Path


def main():
    base = Path.home() / '.local/share/sa-website-monitor'
    settings = base / 'settings.json'
    info = settings.stat()
    if info.st_uid != os.getuid() or info.st_mode & 0o077:
        raise ValueError('Private settings required.')
    config = json.loads(settings.read_text(encoding='utf-8'))
    spec = importlib.util.spec_from_file_location(
        'sa_monitor', base / 'monitor.py'
    )
    monitor = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(monitor)

    trial = dict(config)
    trial['url'] = 'https://sa-connectivity-test.invalid/'
    state, reason = monitor.check_page(trial)
    if state != 'DOWN':
        print('STOP: the deliberately invalid target did not fail.')
        return 1
    if not monitor.send_message(
        config,
        'TEST ONLY: the observer detected a failed request.\n'
        'Target: sa-connectivity-test.invalid\n' + reason
    ):
        print('STOP: test alert delivery was not confirmed.')
        return 1
    print('Failure-path test sent. Check Telegram.')

    state, reason = monitor.check_page(config)
    if state != 'UP':
        print('STOP: the real page check is not passing; inspect it.')
        return 1
    if not monitor.send_message(
        config,
        'TEST ONLY: the real page check now passes from the observer.\n'
        + config['url'] + '\n' + reason
    ):
        print('STOP: passing-check message delivery was not confirmed.')
        return 1
    print('Passing-check test sent. Production settings and state unchanged.')
    return 0


if __name__ == '__main__':
    try:
        raise SystemExit(main())
    except Exception:
        print('TEST ERROR: inspect local setup without sharing credentials.')
        raise SystemExit(1)

Run it:

python3 "$HOME/.local/share/sa-website-monitor/test-external.py"

If both parts succeed, Telegram receives two messages marked TEST ONLY: one for the failed request and one for the passing real page check. Your saved settings and last-notified state stay unchanged.

A manual success is not a scheduled success

On the observer, find Python’s full path:

command -v python3

Prepare the log before cron redirects output into it:

(
  umask 077
  touch "$HOME/.local/share/sa-website-monitor/monitor.log"
  chmod 600 "$HOME/.local/share/sa-website-monitor/monitor.log"
)
crontab -e

Keep existing entries. Add this line to your own crontab, replacing /usr/bin/python3 if your machine printed another path:

*/5 * * * * /usr/bin/python3 "$HOME/.local/share/sa-website-monitor/monitor.py" >> "$HOME/.local/share/sa-website-monitor/monitor.log" 2>&1

After a scheduled run, inspect the recent records:

tail -n 20 "$HOME/.local/share/sa-website-monitor/monitor.log"

The program writes UTC timestamps. Compare them with the current UTC time rather than assuming they match your local clock:

date -u

Repeated UP records without repeated notifications are expected. An old log is not proof that the site stayed healthy; it may mean the schedule stopped running.

Silence needs its own explanation

The external observer is still a machine. It can lose power, stop running cron or lose access to Telegram. A log stored only on that observer cannot notify you when the observer is gone.

A separate missed-heartbeat check can cover part of this gap. The observer reports that a check cycle completed; an independent receiver alerts if those reports stop arriving.

For example, a five-minute check could have a fifteen-minute missing-heartbeat threshold to allow some delay. That is a design choice, not a guarantee. Choose the interval and grace period around the failures and interruptions you can tolerate.

Keep a short record of what you tested

Keep the log under a retention policy. Repeat the relevant tests after changes to the bot token, settings, scheduler, observer host or notification route.

Before you finish

1. Why is a second container on the website VPS insufficient for machine failure?
2. Why check the public hostname rather than localhost?
3. What does the invalid-target test leave unchanged?
4. Why can the real site be healthy while a scheduled log stays old?
5. What is still missing if observer heartbeats are not configured?