Running a Feed Aggregator on Cloudflare Pages With a Twice-Daily Rebuild

A static site only changes when you rebuild it. For a blog that's fine, since you rebuild when you publish. For an aggregator it's the whole problem. The content belongs to 114 other sites, and they publish whenever they like.

Syndicator solves that the boring way. It rebuilds on a timer. Here's the setup behind syndicator.net, which costs nothing to run.

Hosting

The site is a Git-connected Cloudflare Pages project. Every push to the main branch triggers a build. The build command is a short shell script:

node build.mjs
(cd blog && go run .)
rm -rf _site/blog
cp -r blog/dist _site/blog

First the aggregator builds the main site into _site. Then a small Go engine builds this blog, and the result is copied into _site/blog. One project, one deploy, two generators.

There's no framework preset to pick, because there's no framework. Node and Go are both available in the Pages build image, so nothing gets installed.

The timer

Cloudflare Pages has deploy hooks. You create one in the project settings and get a secret URL. An empty POST to that URL starts a fresh build of the branch.

So the scheduler only needs to send one HTTP request, and GitHub Actions does that on a cron:

on:
  workflow_dispatch:
  schedule:
    - cron: "0 2,14 * * *"

That's 02:00 and 14:00 UTC, every twelve hours. The job sends a POST with curl to the hook URL, which sits in a repository secret, and retries three times if the request fails. If the secret is missing, the job logs a warning and exits cleanly instead of failing every run.

The workflow_dispatch line adds a button in the Actions tab for a manual rebuild.

Two notes on GitHub's cron. It runs in UTC, and it's not punctual. Scheduled runs can start several minutes late when GitHub is busy, especially on the hour. For a twice-daily refresh that doesn't matter. For anything time sensitive, pick an odd minute and don't rely on the exact time.

Why twelve hours

It's a trade-off between freshness and load. Every rebuild fetches 114 feeds. At twelve hours, the front page is never more than half a day behind, and each source sees two polite requests a day from the aggregator. Most of the sites in the network post daily or less often, so faster rebuilds would mostly fetch unchanged feeds.

If the sources were news sites, the right answer would be hourly, plus conditional requests so unchanged feeds cost almost nothing.

Cache settings that fit the schedule

Syndicator keeps each fetched feed on disk for 45 minutes. That must be shorter than the rebuild interval, or a persisted cache would serve the previous build's items. At twelve hours there's lots of room.

On the output side, a _headers file tells Cloudflare how long browsers and readers may cache things. The merged feed gets 30 minutes:

/feed.xml
  Cache-Control: public, max-age=1800

CSS and JavaScript get a year with immutable, which is safe because their URLs carry a hash of the file content. Change the stylesheet and the URL changes with it.

What it costs

Cloudflare Pages' free tier covers a site like this with room to spare. Builds are short. The GitHub Actions job runs for a few seconds twice a day, which is about an hour a month of billed minutes, since GitHub rounds each run up to a full minute. No server, no database, nothing to patch.

The same pattern works on Netlify, which has build hooks that take the same empty POST. Syndicator ships with a netlify.toml for that case.

When to use something else

A timed static rebuild is the right tool when a few hours of delay is fine and the sources are polite, cacheable feeds. If you need minute-level freshness, or per-user views, you want a long-running service with a database. For a public front page across a group of sites, the static version is hard to beat: cheap, fast, and nothing to keep running.