How Syndicator Keeps One Broken Feed From Emptying the Front Page

An aggregator depends on dozens of servers it doesn't control. At 114 feeds, something is always a little broken. A site is down for maintenance. A feed times out. A plugin update starts emitting junk. So the useful question is what a build should do when a feed fails, because one usually will.

Syndicator handles that with a few rules. None of them are clever. They just make sure a problem in one feed stays in that feed.

Five per feed, then merge

Each feed contributes at most its five newest items. Only then do all the items get merged and sorted by date.

Without the cap, one busy site that posts thirty times a day would take over the front page, and a quiet site that posts weekly would never show up. With it, every source gets the same small slot, and the merged stream reflects what's new across the group instead of who posts the most.

The Atom output at /feed.xml is set to 1,000 entries, comfortably above five times 114, so every site's latest items make it into the feed even on a slow week.

Bounded fetching

Feeds are fetched by a pool of eight workers. Each request has a 15 second timeout and one retry. So one hanging server can't stall the whole build, and the build doesn't open 114 connections at once and look like an attack to anyone's firewall.

Every request identifies itself with a user agent that names the aggregator and links to its home page. Site owners who see the traffic in their logs know who it is.

Last known good

When a feed fails, Syndicator doesn't just drop the site. After every successful fetch it saves that feed's items to a state file. If the next fetch times out, returns a 404 or can't be parsed, the build falls back to the saved items and logs a warning naming the site and how old the copy is.

After four days without a successful fetch, the site's entry on the sites page gets a "last seen" badge, so the problem is visible without anyone reading a log.

A feed that has never once worked contributes nothing and gets logged as an error. That's the case the fallback can't save. It's also the most common real failure, usually a wrong feed URL. One site in the list had exactly that until this week.

This only helps on a host that keeps the cache folder between builds. On a build platform that starts every build from a clean checkout, the state file is gone each time and a failed feed simply shows nothing until it recovers. Persisting that folder is on the list.

Refuse to publish nothing

If a build collects zero articles in total, it stops with an error instead of deploying. When that happens it almost always means the build machine lost its network, not that 114 sites failed at the same moment. Publishing an empty site in that situation would replace a working page with a blank one. Failing the build keeps the last good deploy online.

Dates that make sense

Sorting by date only works if dates parse. The parser tries pubDate, published, updated, date, dc:date and issued in that order, and JSON Feed's date_published or date_modified. An item with no usable date gets the build time, which puts it at the top for one cycle rather than hiding it.

That last rule has a downside. A feed with no dates at all would sit at the top after every rebuild. None of the 114 do that, but it's worth knowing if you run Syndicator on your own list.

The general lesson

The pattern applies to any system that reads from many sources it doesn't own: cap each source, time out fast, keep the last good copy, and make staleness visible. Most of the work is deciding what to show when things go wrong, because something always does.