WIP: Draft: Address database performance hotspots #7877

lambadalambda · 2026-04-29T11:54:59Z

lambadalambda commented

2026-04-29 11:54:59 +00:00

Summary

Paginate ActivityPub follower/following collections and endorsements in SQL instead of materializing large result sets.
Batch Mastodon view lookups/count queries and use incremental follow counter updates to avoid per-row database work.
Add directory query guardrails/indexes and make stats refresh interval configurable.

## Summary - Paginate ActivityPub follower/following collections and endorsements in SQL instead of materializing large result sets. - Batch Mastodon view lookups/count queries and use incremental follow counter updates to avoid per-row database work. - Add directory query guardrails/indexes and make stats refresh interval configurable.

lambadalambda added 7 commits

2026-04-29 11:55:00 +00:00

Paginate ActivityPub follower/following collections in SQL 08dcf4e095

Moves pagination into the database for follower and following federation
collections, avoiding full materialization of large result sets in memory.

- UserView now fetches only the requested page (10 items) via limit/offset
- Total counts reuse the cached user.follower_count/user.following_count
- The query is skipped entirely when neither items nor counts will be shown
- CollectionViewHelper.collection_page_offset/5 accepts pre-paginated items
  when an explicit total is provided
- Fixes User.get_followers_query/1 to filter out deactivated users,
  matching get_friends_query/1 and follower_count behavior

Make stats refresh interval configurable 16d148f542

The peer list query in calculate_stat_data/0 is expensive on instances
with many remote users because it aggregates split_part(nickname, '@', 2)
across the entire remote user population every refresh cycle.

This change makes the refresh interval configurable via
config :pleroma, :stats, :interval so operators can trade freshness for
lower steady-state DB load. The default remains 60 seconds.

Add directory composite indexes and offset cap f56b11b581

The profile directory endpoint did not have indexes matching its actual
filter and sort pattern, causing expensive parallel seq scans on large
instances.

- Adds partial composite indexes on users for both directory orderings:
  - users_directory_last_status_at_index for the default active path
  - users_directory_id_index for the order=new path
- Caps directory offset at 1000 to prevent expensive deep pagination
- Both indexes use CONCURRENTLY to avoid locking the users table

Batch notification and conversation view lookups f3d98465c5

- NotificationView reuses the precomputed relationship dictionary for the
  is_muted flag instead of calling User.mutes?/2 per notification
- ConversationView preloads conversations and recipients for the whole
  list before rendering, and batch-fetches last activities with objects
  in a single query instead of per participation

Use incremental counter updates for local users on follow/unfollow 0151591da0

The follow/unfollow write path previously recomputed follower and following
counts with aggregate queries after every relationship change. On large
accounts this adds unnecessary write-path latency.

- Local users now get increment/decrement counter updates on the hot path
- Remote users preserve existing behavior (recount or remote fetch)
- User.update_follower_count/1 and update_following_count/1 are kept
  as repair functions for drift correction

Batch self-account count queries in AccountView 67262fb988

Rendering the current user's own account previously fired three separate
aggregate queries for follow requests, unread conversations, and unread
notifications. These are now computed concurrently via Task.async and
reused across the three maybe_put_* helpers.

- fetch_self_counts/1 runs counts in parallel when self=true
- New maybe_put_* clauses read precomputed counts from opts[:self_counts]
- Fallback clauses preserve behavior for callers that don't pass self_counts

Paginate endorsements endpoints

ci/woodpecker/pr/woodpecker Pipeline failed

Details

ede7354ac8

Both GET /api/v1/accounts/:id/endorsements and GET /api/v1/endorsements
previously used Repo.all(), which could return unbounded result sets.
They now use Pagination.fetch_paginated/2 with Mastodon-compatible link
headers, matching the existing pattern for mutes, blocks, and other
paginated list endpoints.

lambadalambda changed title from ~~Draft: Address database performance hotspots~~ to WIP: Draft: Address database performance hotspots

2026-04-29 11:58:41 +00:00

lambadalambda added 1 commit

2026-04-29 12:42:18 +00:00

Fix stale follow counters in AP collections

ci/woodpecker/pr/woodpecker Pipeline failed

Details

6f8ff75501

mkljczk reviewed

2026-05-01 10:30:27 +00:00

lib/pleroma/web/mastodon_api/views/account_view.ex

					
				@ -349,3 +356,3 @@

				    |> maybe_put_chat_token(user, opts[:for], opts)

				    |> maybe_put_activation_status(user, opts[:for])

				    |> maybe_put_follow_requests_count(user, opts[:for])

				    |> maybe_put_follow_requests_count(user, opts)

are those actually used by clients like pleroma-fe?

ci/woodpecker/pr/woodpecker Pipeline failed

Details

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/database-performance-hotspots:fix/database-performance-hotspots

git switch fix/database-performance-hotspots

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch develop

git merge --no-ff fix/database-performance-hotspots

git switch fix/database-performance-hotspots

git rebase develop

git switch develop

git merge --ff-only fix/database-performance-hotspots

git switch fix/database-performance-hotspots

git rebase develop

git switch develop

git merge --no-ff fix/database-performance-hotspots

git switch develop

git merge --squash fix/database-performance-hotspots

git switch develop

git merge --ff-only fix/database-performance-hotspots

git switch develop

git merge fix/database-performance-hotspots

git push origin develop

Sign in to join this conversation.

No reviewers

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

pleroma/pleroma!7877

Rows
Columns