The Telegram bot that runs my server, and the one I deleted

I built a Telegram bot to run three things: a VPS that does the work Vercel times out on, an X posting queue, and the outreach CRM. It is still running exactly one of them. The other two were removed on purpose, and one of those removals is a destructive migration sitting in this repo with a comment explaining itself.

So this is a build guide with a verdict attached. The building part is genuinely small: a bot is one HTTPS handler and a token, and you can have a working one before your coffee is cold. The part worth writing down is which of your systems should be allowed to have one, because the failure mode is not that the bot breaks. It is that the bot becomes a second place where the truth lives.

A bot is one handler and a token

You get a bot by messaging @BotFather, which hands back a token shaped like 8123456789:AAH…. That token is the entire credential. It goes in the URL of every call you make, which means it ends up in shell history, in process listings, and in any log that records outbound URLs. Treat it like a private key that you are contractually obliged to put in a query string.

Then you choose how updates reach you, and there are two ways. They are mutually exclusive by design: while a webhook is set, getUpdates returns nothing at all.

• getUpdates

  Long polling

  outbound only · no port · no certificate

  Your process asks Telegram for updates and holds the connection open until there are some. Nothing on the internet can reach it, because nothing is listening. The transport everybody calls the beginner one is the only one a box behind a private network can use at all.

• setWebhook

  Webhook

  inbound HTTPS · ports 443, 80, 88, 8443

  Telegram POSTs each update to a public URL of yours. Zero idle cost, which is what you want on a platform billing per invocation, and it needs a real hostname with a real certificate on one of four ports. There are no others.

Setting the webhook is one call, and the two parameters that matter are the ones nobody sets on the first attempt.

curl -sS "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/setWebhook" \
  -d url="https://example.com/api/telegram" \
  -d secret_token="$TELEGRAM_WEBHOOK_SECRET" \
  -d allowed_updates='["message","callback_query"]' \
  -d drop_pending_updates=true

# When it is quiet and you do not know why, this is the only
# debugger: pending_update_count and last_error_message say what
# Telegram currently thinks of you.
curl -sS \
  "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo"

allowed_updates is a subscription, not a filter you apply later: leave it off and you are paying an invocation to throw away edited-message and reaction events forever. drop_pending_updates matters on the day you fix a crash loop, because Telegram has been queueing everything you failed to answer and will deliver all of it the moment you come back up.

The four lines of security that actually matter

Here is the mistake that ships most often. The token authenticates you to Telegram. It authenticates nothing in the other direction. Your webhook URL is a public endpoint accepting unauthenticated JSON, and an attacker who guesses it can hand your bot any update they like, including one that looks exactly like you pressing a button.

Two checks close that, and they are different checks. The secret token header proves the request came from Telegram. The chat allowlist proves it came from you, which is a separate fact, because anyone who finds your bot can press /start on it.

// app/api/telegram/route.ts
import { timingSafeEqual } from 'node:crypto'

const SECRET = process.env.TELEGRAM_WEBHOOK_SECRET!
// Numeric chat ids, comma separated. In my case, one entry: mine.
const ALLOWED = new Set(process.env.TELEGRAM_CHAT_IDS!.split(','))

// Constant time, and length-checked first: timingSafeEqual throws
// outright on a length mismatch, which would turn a wrong guess into
// a 500 instead of a 401.
function authentic(given: string) {
  const a = Buffer.from(given)
  const b = Buffer.from(SECRET)
  return a.length === b.length && timingSafeEqual(a, b)
}

export async function POST(req: Request) {
  const given = req.headers.get('x-telegram-bot-api-secret-token')
  if (!authentic(given ?? ''))
    return new Response(null, { status: 401 })

  const update = await req.json()
  const chatId = String(
    update.message?.chat?.id ??
      update.callback_query?.message?.chat?.id ??
      ''
  )

  // Telegram vouched for the sender. Nobody vouched for who they are.
  // 200 and not 403, deliberately: a rejection Telegram reads as a
  // failure is a rejection it will deliver to you again, forever.
  if (!ALLOWED.has(chatId)) return new Response(null, { status: 200 })

  await handle(update)
  return new Response(null, { status: 200 })
}

That last line is not a style note. It is the first race condition in the post, and it is the one that generates traffic while you sleep.

Build a Telegram Bot Webhook Setup: Step by Step 2025 TutorialThe Web Mastero · loads YouTube on click

Every update arrives at least once

Telegram does not fire and forget. An update you do not answer with a 2xx stays in your queue and comes back, which is the right behaviour and the source of nearly every duplicate-action bug people write about. Your handler is not called once per event. It is called at least once per event, and the difference is a delivery guarantee you have to hold up your end of.

Two consequences, in order. Answer fast, then work: your response is what closes the delivery, so a handler that runs a thirty-second job before returning 200 has told Telegram it failed, and Telegram will helpfully start the job again. And make the work idempotent, because even a fast handler can be redelivered when the network eats the response rather than the request.

Idempotency here is free, because every update carries a unique update_id. You do not need a lock or a queue. You need a primary key.

-- The insert IS the dedupe. A redelivered update conflicts and does
-- no work, and it is the database deciding that, not two code paths
-- agreeing to.
create table tg_updates (
  update_id bigint primary key,
  seen_at   timestamptz not null default now()
);

const { rowCount } = await sql`
  insert into tg_updates (update_id) values (${update.update_id})
  on conflict do nothing`

// Seen it. Answer 200 so Telegram stops asking, and do nothing else.
if (rowCount === 0) return

The button you tap twice

Inline keyboard buttons are where a bot stops being a notifier and starts being a controller, and they are where the concurrency gets real. A tap arrives as a callback_query update. The button does not visually change when you press it: it shows a small loading state and keeps it until you call answerCallbackQuery. So on a slow connection, the honest user behaviour is to tap it again.

Now you have two updates, with two different update_ids, expressing one intention. The dedupe table above will not save you, correctly, because these genuinely are two different events. What saves you is refusing to write the decision in two steps.

The broken version reads the row, checks the status in your application, and then writes. Between the read and the write there is a window, and the second tap lives in it. The fixed version never opens the window: the condition and the write are the same statement, and the database settles it.

// One statement. The WHERE clause is the lock: whichever tap arrives
// first changes the row; the second matches nothing and does nothing.
const claimed = await sql`
  update jobs set status = 'running', started_at = now()
  where id = ${jobId} and status = 'queued'
  returning id`

// Zero rows back is a SUCCESS, not an error: somebody already did this,
// which is exactly what the user wanted. Say so, and never throw. An
// unanswered callback leaves the button spinning until the client
// gives up.
await answerCallbackQuery(
  query.id,
  claimed.length ? 'Started' : 'Already running'
)

This is the same shape the X publisher on this site uses, one layer down. Its claim is a plpgsql function holding a transaction-scoped advisory lock, so the daily cap, the sixty-minute gap and the write are one indivisible step and two overlapping cron runs cannot both decide they are the only publisher. A row stuck mid-flight for ten minutes is released to failed, never back to approved: the post may have landed before the process died, and a queue that retries an unknown outcome is a queue that double-posts.

Different mechanism, identical principle. If two things can want the same row, exactly one statement is allowed to decide.

The VPS half, which is the part I kept

Long jobs do not belong on Vercel. Its functions time out, so scraping, batch enrichment and model calls at volume run on a Hetzner box that writes its results to Postgres, and the cockpit pages read them. That box sits behind Tailscale and has no public ingress, which is the whole point of it.

Which leaves a real problem: it has no interface. Giving it a web UI means giving it a public door, a hostname, a certificate and a login page, all so that one person can occasionally type how much disk is left. SSH from a phone is technically possible and nobody does it twice.

A polling bot is the answer, and the reason is precisely the property that makes polling look inferior on paper. The process opens an outbound HTTPS connection to Telegram and holds it. There is no listening socket, no inbound port, no reverse proxy, no certificate to renew and no hostname to discover. The attack surface of the control plane is one outbound connection to one host. You cannot get that from a webhook at any price.

Run it as a service, not in a terminal

A bot started in a shell dies with the shell. systemd is already on the box, it restarts the process, it captures the logs, and it is where the sandboxing lives.

# /etc/systemd/system/ops-bot.service
[Unit]
Description=Ops bot
After=network-online.target
Wants=network-online.target

[Service]
User=opsbot
ExecStart=/usr/bin/node /srv/ops-bot/index.js
# chmod 600. The token lives in this file, never in the unit.
EnvironmentFile=/etc/ops-bot.env
Restart=always
RestartSec=5

# It runs a fixed list of jobs. It needs nothing else on this machine.
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/srv/ops-bot/state

[Install]
WantedBy=multi-user.target

Restart=always plus RestartSec=5 is not a detail. Telegram holds your poll open for up to a minute at a time and the connection will occasionally break for reasons that are nobody’s fault. A bot that treats a dropped long poll as fatal is a bot that is down every few days and appears to be up.

A menu, not a shell

The temptation, once the bot is running as a user on your server, is to let it run whatever you type. Do not. Not because you do not trust yourself, but because the allowlist above is the only thing between a leaked chat id and a root shell, and defence in depth means the second layer assumes the first one failed.

// A command names a job. It never carries one. Nothing a message
// contains ever reaches a shell, because there is no shell here.
const JOBS = {
  enrich: ['node', '/srv/jobs/enrich.mjs'],
  backup: ['/usr/local/bin/backup.sh'],
  disk:   ['df', '-h'],
} as const

const job = JOBS[command as keyof typeof JOBS]
if (!job)
  return send(chatId, `Unknown. Try: ${Object.keys(JOBS).join(', ')}`)

// spawn with an argument array, never exec with a string: nothing is
// there to interpret a semicolon, so a semicolon is just a character.
spawn(job[0], job.slice(1), { stdio: ['ignore', 'pipe', 'pipe'] })

Do not stream a log into a chat

The first version of anything like this pipes job output straight into sendMessage, and the first job with a chatty log takes the bot down. Telegram’s documented limits are about one message per second to a single chat, roughly thirty messages per second overall, and no more than twenty a minute into a group. A build log clears all three in the first second and earns a 429 with a retry_after you then have to honour.

The fix is one message that changes. Send a placeholder, buffer the output, and call editMessageText on a timer of a second or two with the last twenty lines. One message per job instead of one per line, a chat you can still scroll a week later, and a rate limit you never touch.

The CRM half: alerts out, one switch in

The outreach engine has four things it ever needs to tell me: the daily cap tripped, someone replied, someone opted out, the kill switch fired. That is a handful of messages a week.

I did not build a bot for it, and I want to be plain about why, because the reasoning is the useful part. Those alerts already go to the inbox that receives replies, and to a web push on my phone. Adding a bot token, a webhook and an inbound surface to deliver four messages a week is a dependency that buys nothing. The correct number of moving parts for a notification is the number you already have.

There is one thing on that list a bot would genuinely improve, and it is the kill switch. Not because it is frequent, but because it is the one action whose value is entirely in how fast you can take it from wherever you are standing. If you build that, build it as the same claim as everything else in this post: one conditional update on the settings row, not a command that reads a value and writes it back.

And the alert path itself must never be able to break the thing it is alerting about. The notifier on this site never throws, on purpose, and swallows its own failures. An alert that can fail a send is strictly worse than a missed alert, because the first one costs you the work and the second one costs you the message.

The X half, which I deleted

This is the part I got wrong, and it was the part I was proudest of.

The X queue drafts posts and every one of them needs a human decision. The bot pushed each draft into a chat as a card carrying three buttons: ✅, ❌, ✏️. Approving from a locked phone was two taps. It was genuinely delightful, and it lasted until I looked at what it had quietly become.

A card in a chat is a snapshot. The draft is a row in a database. Edit the draft in the cockpit and the card does not change: it is now a stale copy of the post, in a chat, with a working approve button under it. To fix that you write reconciliation code for a chat message. To find the right card you keep a telegram_msg_id column on every draft. To handle an edit typed as a loose reply you keep an awaiting_edit flag, which is conversational state, which is another word for a state machine living in a chat window.

The deeper problem is that it made a second controller. The rails that matter here are five posts a day, one per window, sixty minutes apart, and they are enforced in one plpgsql function. Two things able to move a draft to approved means those rails have to hold under both, and the way to guarantee a second controller never diverges is for it not to exist, rather than for it to be carefully kept in line.

So it went, and the columns went with it, in a migration that says so out loud.

-- The last of the Telegram bot.
--
-- DESTRUCTIVE, and deliberately its own migration so that is
-- unmistakable. Idempotent: `if exists` makes a re-run a no-op.

alter table x_drafts drop column if exists telegram_msg_id;
alter table x_drafts drop column if exists awaiting_edit;

What replaced it is a web push notification carrying the post itself, so it reads on the lock screen, and a tap that opens the queue with approve one more tap away. I will not pretend that is a straight upgrade. iOS renders no actions on a web push, whatever the payload says, so a lock-screen approve was never on offer once the bot was gone. The honest accounting is that I paid one tap.

For one tap I removed a bot token, a public webhook, an inbound surface to authenticate, a state machine in a chat window, two database columns, and a second copy of every draft that could disagree with the row it came from. The cockpit is the only controller again, and the phone went back to being a notification rather than a system of record.

If you are building one, in order

1. 01Decide the transport before you write a line. Private box with no public door: long polling. Serverless host billing per invocation: webhook. Both, in one system: two bots, because one bot cannot do both.
2. 02Set secret_token on setWebhook and compare it in constant time. The bot token proves you to Telegram and proves nothing to you.
3. 03Allowlist chat ids, and answer a stranger with 200 rather than 403, so a rejection is not something Telegram retries forever.
4. 04Insert update_id into a table with a primary key and return early on conflict. Every update arrives at least once, and this is the entire fix.
5. 05Answer the webhook first, then do the work. Your 200 is what closes the delivery, not what reports the outcome.
6. 06Make every button press one conditional UPDATE with a WHERE on the current state. Zero rows affected means someone got there first, and that is a success to report, not an error to throw.
7. 07Call answerCallbackQuery on every path, including the losing one, or the button spins until the client gives up.
8. 08On a VPS: systemd with Restart=always, a fixed map of allowed jobs, spawn with an argument array, and one edited message instead of a stream of new ones.
9. 09Send plain text or HTML. MarkdownV2 will reject your stack trace at the moment you most need to read it.

The rule I ended on

A bot is a remote control. It stops being one the moment it holds a copy of something, and everything expensive I have described happened after that line was crossed.

So give it the machine that has no other door, where a control plane that makes only outbound connections is not a compromise but the best available design. Keep it off the surface that already has a controller, where a card in a chat is a second copy of a row and every hour it exists is an hour it can be wrong. And whatever it is allowed to write, make that write one statement, so that tapping twice and tapping once end in the same place and the button can tell you honestly which one it was.