buildlog

An RSS digest that arrives by email

feedpost

I follow about forty feeds and kept forgetting to open the reader. Email I check anyway, so the feeds now come to me once a day as a single message.

Project
feedpost
Stack
Python, SQLite, SMTP
Time spent
About 5 evenings
Status
sends every morning

didWhat I did

The script reads a list of feed URLs, fetches each one, and keeps a small SQLite table of items it has already sent. Anything new goes into the day's email, grouped by feed, with the title, the link and the first two sentences of the summary.

CREATE TABLE IF NOT EXISTS seen (
    feed_url  TEXT NOT NULL,
    item_key  TEXT NOT NULL,
    first_seen TEXT NOT NULL DEFAULT (datetime('now')),
    PRIMARY KEY (feed_url, item_key)
);

The email goes out as plain text with a simple HTML part, through the SMTP server of my mail provider. The password lives in an environment variable loaded from a file only my user can read. It runs from cron:

30 6 * * *  cd ~/feedpost && ./venv/bin/python digest.py >> digest.log 2>&1

A typical morning is 15 to 30 items, which takes about five minutes to skim. That is the whole point: one sitting, then done.

Morning digestSun 15 Mar, 06:30

23 new items from 11 feeds

sqlite-notes.example.org 3
Why WAL mode is not the default
homelab.example.net 2
Moving cron jobs to systemd timers
gopher-diary.example.com 4
Small CLIs and the flag package

Quiet feeds: 1 returned nothing for 3 days

One plain email a day. The quiet-feeds line exists because of a two-week silent failure.

brokeWhat broke

Duplicates. I keyed items on their GUID, which is what the GUID is for. Some feeds don't have one. Others generate a new one every time the post is edited, so a typo fix meant the same article twice. Hashing the title didn't help either, because titles get edited too.

What stuck: use the link, with tracking parameters stripped and the trailing slash normalised. It is not perfect, but in a month it produced no duplicates.

def item_key(entry) -> str:
    url = urlsplit(entry.get("link", ""))
    query = [(k, v) for k, v in parse_qsl(url.query) if not k.startswith("utm_")]
    path = url.path.rstrip("/") or "/"
    return urlunsplit((url.scheme, url.netloc.lower(), path, urlencode(query), ""))

Silent failure. One site started returning an HTML error page with status 200. The parser happily returned zero entries, the digest just had nothing from that feed, and I didn't notice for two weeks. Now any feed that yields zero items three days in a row is listed at the bottom of the email under "Quiet feeds".

Spam folder. The first week's digests went straight to spam on my own account. The sending domain had no SPF or DKIM records. Adding both fixed it the next morning.

nextWhat's next

  • A per-feed limit. One news site posts thirty items a day and drowns everything else.
  • A weekly mode for slow personal blogs, sent on Saturdays, so they don't get lost between busier feeds.
  • Swap cron for a systemd timer with Persistent=true, so a run missed while the machine was asleep still happens when it wakes.