Summary

In a high-volume self-hosted Hook0 deployment, the old events cleanup task can saturate PostgreSQL for a long period of time even when it deletes zero rows.

The cleanup query in api/src/old_events_cleanup.rs appears to be unbounded:

WITH retention AS (
  SELECT
    a.application__id,
    MAKE_INTERVAL(days => COALESCE(LEAST(a.days_of_events_retention_limit, p.days_of_events_retention_limit), $1) + $2) AS events_retention_limit
  FROM event.application AS a
  INNER JOIN iam.organization AS o ON o.organization__id = a.organization__id
  LEFT JOIN pricing.price AS pr ON pr.price__id = o.price__id
  LEFT JOIN pricing.plan AS p ON p.plan__id = pr.plan__id
)
DELETE FROM event.event
WHERE event__id IN (
  SELECT e.event__id
  FROM event.event AS e
  INNER JOIN retention AS r ON r.application__id = e.application__id
  WHERE e.received_at + r.events_retention_limit < statement_timestamp()
);

Because this scans/joins across event.event, it can create severe database pressure independent of whether rows are actually eligible for deletion.

Observed impact

On a production self-hosted deployment handling a large webhook burst:

• The old-events cleanup query ran for about 3,673 seconds (~61 minutes).
• rows_affected = 0.
• PostgreSQL CPU reached 100%.
• Webhook delivery attempts backed up while the database was saturated.
• The oldest pending delivery attempt age grew into the tens of minutes.
• Query Insights showed this cleanup query processing tens of millions of rows.
• Other worker queries, especially request-attempt fetching/updating, became much slower while the cleanup was running.

The issue was especially surprising because OLD_EVENTS_CLEANUP_REPORT_AND_DELETE=false still appears to perform the expensive work inside a transaction and then roll it back. That setting prevents committed deletes, but it does not prevent the expensive scan.

Why current config does not fully mitigate this

Hook0 exposes these relevant settings:

• OLD_EVENTS_CLEANUP_PERIOD_IN_S
• OLD_EVENTS_CLEANUP_GRACE_PERIOD_IN_DAY
• OLD_EVENTS_CLEANUP_REPORT_AND_DELETE
• QUOTA_GLOBAL_DAYS_OF_EVENTS_RETENTION_LIMIT

However, there does not appear to be a setting for:

• disabling old-events cleanup entirely
• cleanup batch size / LIMIT
• max runtime / statement timeout for old-events cleanup
• per-application cleanup
• skipping the expensive delete query when OLD_EVENTS_CLEANUP_REPORT_AND_DELETE=false

Increasing OLD_EVENTS_CLEANUP_PERIOD_IN_S reduces how often the problem can occur, but it does not make the query safe. Running it more frequently is also risky because the query can still scan a large retained dataset even when nothing is eligible for deletion.

Suggested fixes

Some possible approaches:

• Add an explicit config flag to disable old-events cleanup entirely.
• If OLD_EVENTS_CLEANUP_REPORT_AND_DELETE=false, do not run the full delete query; use a cheaper estimate/count or skip cleanup.
• Rewrite cleanup as bounded batches, e.g. delete at most N rows per run.
• Process one application at a time to avoid scanning all applications/events at once.
• Add a statement timeout specific to old-events cleanup.
• Use an indexed cutoff strategy based on received_at where possible.
• Move cleanup out of the API process or make it safe to run as a separate maintenance job.

Expected behavior

Housekeeping should not be able to starve webhook ingestion or delivery workers on high-volume self-hosted deployments. Old event cleanup should be bounded, interruptible, and configurable enough that operators can avoid database-wide load spikes.