Scheduled scripts fail silently; seven checks make them speak up

On January 31, 2017, an engineer at GitLab.com wiped a database directory on the primary server, “errantly thinking” it was the secondary. The storage bucket meant to hold the backups was empty, and GitLab lost about six hours of data. Its backup job, set to run every 24 hours, had been failing, and the alert emails it sent were rejected by the receiving server.

GitLab’s own summary: “we were never aware of the backups failing, until it was too late.” That is the risk in every scheduled script. It runs where nobody watches, so a failure stays silent unless someone builds it a voice.

We think that is where most of the effort in a small automation belongs, and a 34-line script that chases late invoices shows why. The checklist at the end works for any script you schedule.

Two lines of the invoice script do the job

A hypothetical small company’s accounting system drops a CSV export of invoices into a shared folder every night. The columns are number, customer, amount, due and status. Someone opens it each morning, sorts by due date and tells the team about anything late, except on busy days.

The script below does that every weekday at 08:00 and posts to the team chat through an incoming webhook. It runs on Node.js 22 or newer, with no packages.

// overdue.mjs: post overdue invoices from a CSV export to a chat webhook.
// Expects columns number,customer,amount,due,status, with due as YYYY-MM-DD
// and no commas inside fields.
import { readFile } from 'node:fs/promises';

const file = process.argv[2] ?? 'invoices.csv';
const webhook = process.env.WEBHOOK_URL;
if (!webhook) throw new Error('WEBHOOK_URL is not set');

const today = new Date().toISOString().slice(0, 10); // UTC date
const [header, ...lines] = (await readFile(file, 'utf8')).trim().split(/\r?\n/).filter(Boolean);
const cols = header.split(',');
const invoices = lines.map((line) =>
  Object.fromEntries(line.split(',').map((value, i) => [cols[i], value.trim()])),
);

const isDate = (s) => /^\d{4}-\d{2}-\d{2}$/.test(s ?? '');
const open = invoices.filter((inv) => (inv.status ?? '').toLowerCase() !== 'paid');
const overdue = open.filter((inv) => isDate(inv.due) && inv.due < today);
const unclear = open.filter((inv) => !isDate(inv.due));

if (overdue.length === 0 && unclear.length === 0) process.exit(0);

const row = (inv) => `${inv.number}  ${inv.customer}  ${inv.amount}  due ${inv.due || '?'}`;
const parts = [];
if (overdue.length) parts.push(`Overdue (${overdue.length}):`, ...overdue.map(row));
if (unclear.length) parts.push(`Check by hand, no readable due date (${unclear.length}):`, ...unclear.map(row));

const res = await fetch(webhook, {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ text: parts.join('\n') }),
});
if (!res.ok) throw new Error(`Webhook answered ${res.status}`);

Guess how many of its 28 non-blank lines decide which invoices are late. Two: the filters that build open and overdue.

Seven lines exist only because something can go wrong, from the check for a missing secret to the check on the webhook’s answer. Two of those are comments that state assumptions. Five more do the job with a guard built in, so 12 of the 28 deal with failure, including both lines that do the job.

Bad input looks exactly like a quiet morning

Amazon’s S3 outage on February 28, 2017 started with input. At 9:37AM PST an authorized team member following an established playbook ran a command, and “one of the inputs to the command was entered incorrectly.” It removed a larger set of servers than intended, and S3 in the region did not finish recovering until 1:54PM PST. AWS’s fix went into the tool, which now removes capacity more slowly and refuses to take a subsystem “below its minimum required capacity level.”

A small script needs the same habit of refusing input that cannot be right. The CSV format’s specification, RFC 4180, ends each record with a carriage return and a line feed. A file that follows it puts an invisible carriage return before every line break.

We deleted the script’s handling for that character and ran a test file. A paid invoice showed up as overdue, because the status column’s name now ended in the stray character.

Our tests also found a hole in the script itself. A file with only a header row made it exit cleanly and post nothing, which is exactly what it does on a good day. It never checks the file’s date either, so if the nightly export stops, the script keeps reading the last one. Two more refusals, for zero rows and for a file older than a day, would close both holes.

Every assumption needs a place to fail out loud

The same RFC says fields containing commas “should be enclosed in double-quotes”, which the script’s plain split on commas would break. The comment at the top says the export has no such fields. That is fine as long as it stays true, and the written assumption tells the next person where to look when it stops being true.

Dates get the same treatment. Any due date that is not YYYY-MM-DD goes to a “check by hand” list. Without that list, a row where someone typed 30/09 would never count as overdue, and the script would say nothing about it every morning.

The date comparison is plain text. RFC 3339 notes that dates written this way, in a single time zone, “may be sorted as strings”, and toISOString() is always UTC. For a run at 08:00 in a time zone an hour or two ahead of UTC, that is the same calendar day. Just after local midnight it would still be yesterday in UTC, and an invoice due yesterday would wait a day.

Cron has failure modes of its own, and its manuals admit them

Put the webhook URL in a .env file next to the script, and run it once by hand before you schedule anything:

# .env holds one line: WEBHOOK_URL=https://chat.example.com/hooks/your-hook
node --env-file=.env overdue.mjs invoices.csv

# Then add it to cron with crontab -e: weekdays at 08:00
0 8 * * 1-5 cd /srv/overdue && /usr/bin/node --env-file=.env overdue.mjs /data/exports/invoices.csv >> run.log 2>&1

Each part of that cron line answers a failure. Cronie’s cron daemon sets its own PATH rather than inheriting one, and its manual offers a -P flag for the opposite. So the line names /usr/bin/node in full instead of hoping node is found (which node shows where yours is).

The >> run.log 2>&1 keeps every error in a file. Without it, cron mails output to the crontab’s owner, or writes it to the system log when sendmail is not installed. GitLab’s alerts went by mail too.

Then there is the clock, because 0 8 * * 1-5 names a time without a time zone. GitHub Actions runs schedules “in UTC” by default. Kubernetes reads them in the controller’s “local time zone” unless .spec.timeZone is set, and cronie lets a crontab pick one with CRON_TZ. Run date on the server before you trust that 08:00 means your 08:00.

Daylight saving time is worse, because cronie’s two manuals disagree. Its crontab page says jobs in the “missing hours” are “not to be run” and jobs in a repeated hour run twice. Its daemon page says skipped jobs “will be run immediately” and double runs are avoided. Whichever is right, we would schedule nothing in the hour the clocks change.

A scheduled job can run twice, or not at all

Cron assumes the machine is always on. That assumption is why anacron exists: its manual says that “unlike cron(8), it does not assume that the machine is running continuously.”

Bigger schedulers say so outright. Kubernetes creates a job “approximately once” per scheduled time, because “two Jobs might be created, or no Job might be created.” Its advice is that jobs “should be idempotent”, which means safe to run twice.

GitHub Actions warns that under heavy load “some queued jobs may be dropped”. In a public repository, it turns schedules off after 60 days without activity.

For the invoice script, a double run posts the same list twice, which is annoying and harmless. A script that sends invoices or charges cards cannot shrug it off. It needs a record of what it already did. In our view, that record is the point where a script should grow into a service.

Silence needs an alarm that lives somewhere else

On a bad morning the team sees this message. The body below is what our test run sent:

{
  "text": "Overdue (2):\n1042  Acme Freight  12400  due 2026-09-24\n1047  Example Studio  3150  due 2026-09-30\nCheck by hand, no readable due date (1):\n1051  Harbor Print  8900  due 30/09"
}

On a good morning it sees nothing. It would see the same nothing if the server had died three weeks ago. Google’s SRE book says monitoring should answer “what’s broken, and why?”, and that “it’s better to spend much more effort on catching symptoms than causes.” For a scheduled script the symptom is simple: it did not finish, and one alert on that covers every cause at once.

That alert is a dead man’s switch: the job checks in after every success, and silence sets off the alarm. Healthchecks.io, one such service, says it “works as a dead man’s switch for processes that need to run continuously or on a regular, known schedule.” Its free Hobbyist plan covers 20 jobs.

Its cron guide adds one command after the job: && curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/your-uuid-here. The && sends the ping only when the job exits with code 0. If no ping arrives within the grace time, the check goes down and the service sends its alerts.

This is why the script throws instead of logging and carrying on. An uncaught error makes Node exit with code 1, so no ping goes out. We tested it with no WEBHOOK_URL set, and with a webhook that answered 410: both runs exited with 1. Slack’s webhooks answer with errors such as channel_is_archived when a channel is archived, which the res.ok check turns into that exit code.

The alarm must not share the script’s fate. During the S3 outage, AWS could not update its own status dashboard “because of a dependency the SHD administration console has on Amazon S3.” A check that runs on the same server as the script dies with it.

What this means for you

The decision is how much failure handling a small script deserves. Too little and it fails like GitLab’s backups; too much and the alerts turn to noise. The SRE book warns that when pages come too often, “employees second-guess, skim, or even ignore incoming alerts.” Its rule is that “every page should be actionable”, and we would hold a chat channel to the same rule.

So aim for two kinds of message. One says the data needs a person, like the “check by hand” list; the other says the script did not run. Everything else goes to the log.

Before you schedule a script, run it past this list:

1. The assumptions are written at the top: columns, formats and time zone.
2. Bad input stops the run or goes to a person: zero rows, a stale file, a missing column, a value that cannot be read.
3. Every failure exits with a non-zero code, including an error answer from any service the script calls.
4. Secrets live outside the file. Slack says a webhook URL “contains a secret”, and Healthchecks.io says to treat the IDs in its ping URLs as secrets too.
5. The clock is pinned. You have run date on the server, set the time zone where the scheduler allows it, and kept clear of the hour the clocks change.
6. A double run is harmless, or the script records what it already did.
7. A check-in runs on another machine, with a grace time longer than the job takes.

GitLab’s response in 2017 points the same way. Among its follow-up work it listed “Prometheus monitoring for backups”, and it started working on a public backup monitoring dashboard. The backup script was never the hard part. Knowing whether it had worked was.

Sources

• Postmortem of database outage of January 31, GitLab, 2017
• Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region, AWS, 2017
• Monitoring Distributed Systems, Rob Ewaschuk, in Google’s Site Reliability Engineering
• Healthchecks.io documentation, cron guide and pricing
• cronie manual pages: crontab(5) and cron(8); anacron(8)
• CronJob, Kubernetes documentation
• Events that trigger workflows: schedule, GitHub Docs
• RFC 4180, CSV format, and RFC 3339, date and time on the internet
• Date.prototype.toISOString(), MDN
• process: uncaughtException, Node.js documentation
• Sending messages using incoming webhooks, Slack