Fix four query hotspots found on a large instance #7982

Open
lambadalambda wants to merge 4 commits from fix/query-hotspots into develop

Four independent fixes, one commit each, from profiling a 312 GB instance (107M activities, 1M users) with pg_stat_statements. Companion to #7981.

  1. Stats peers list hourly instead of every minute. distinct split_part(nickname, '@', 2) over every remote user took 2.7 s and ran 1440 times a day: 3.7M seconds total, the single largest consumer of database time on that instance. Counts still refresh every minute; force_update/0 still recomputes everything.
  2. tag_reject as an anti-join. object.id not in subquery(...) becomes a hashed subplan whose build cost the planner charges per rescan of the objects lookup (same mechanism as #7981). A correlated NOT EXISTS is planned as an anti-join probing the hashtags_objects primary key. Unknown tag names no longer add a filter at all.
  3. Single hashtag compared with =. With hashtag_id = ANY(array) PostgreSQL 15 cannot walk the (hashtag_id, object_id) primary key in order, so it walks the global object_id index newest-first and filters every tagged object on the instance. Measured on a 59k-object tag: 257k rows filtered vs 514, 129 ms vs 50 ms warm; cold the difference is far larger.
  4. URL searches match exactly. A URL query OR'ed its fragments through the text index, so de or org matched tens of thousands of accounts that were then all trigram-ranked, 1-2 s per search. URLs now match by resolution, ap_id, uri and the profile URL's name@host nickname (which is what the IDNA test relied on). Two tests dropped a second result that was a text-search artifact (com_user matching com:*).

Not addressed here, noticed while at it: URL matches bypass the block/invisible/unreachable filters via or_where in select_top_users. Pre-existing, worth its own issue.

Tests: stats, user search, activity_pub, timeline/search/instance controller suites pass.

Four independent fixes, one commit each, from profiling a 312 GB instance (107M activities, 1M users) with pg_stat_statements. Companion to #7981. 1. **Stats peers list hourly instead of every minute.** `distinct split_part(nickname, '@', 2)` over every remote user took 2.7 s and ran 1440 times a day: 3.7M seconds total, the single largest consumer of database time on that instance. Counts still refresh every minute; `force_update/0` still recomputes everything. 2. **`tag_reject` as an anti-join.** `object.id not in subquery(...)` becomes a hashed subplan whose build cost the planner charges per rescan of the objects lookup (same mechanism as #7981). A correlated `NOT EXISTS` is planned as an anti-join probing the `hashtags_objects` primary key. Unknown tag names no longer add a filter at all. 3. **Single hashtag compared with `=`.** With `hashtag_id = ANY(array)` PostgreSQL 15 cannot walk the `(hashtag_id, object_id)` primary key in order, so it walks the global `object_id` index newest-first and filters every tagged object on the instance. Measured on a 59k-object tag: 257k rows filtered vs 514, 129 ms vs 50 ms warm; cold the difference is far larger. 4. **URL searches match exactly.** A URL query OR'ed its fragments through the text index, so `de` or `org` matched tens of thousands of accounts that were then all trigram-ranked, 1-2 s per search. URLs now match by resolution, ap_id, uri and the profile URL's `name@host` nickname (which is what the IDNA test relied on). Two tests dropped a second result that was a text-search artifact (`com_user` matching `com:*`). Not addressed here, noticed while at it: URL matches bypass the block/invisible/unreachable filters via `or_where` in `select_top_users`. Pre-existing, worth its own issue. Tests: stats, user search, activity_pub, timeline/search/instance controller suites pass.
The peers query scans every remote user; on a 1M-user instance it took
2.7 s and ran 1440 times a day, the single largest consumer of database
time. Counts still refresh every minute.
Same planner problem as the notification fix: the hashed subplan's
build cost is charged per rescan of the objects lookup. A correlated
NOT EXISTS is planned as an anti-join probing hashtags_objects by key.
With `hashtag_id = ANY(array)` the planner cannot use the (hashtag_id,
object_id) primary key for an ordered scan and instead walks the global
object_id index newest-first, filtering every tagged object on the
instance. Measured on a 59k-object tag: 257k rows filtered vs 514.
fix(search): match URL queries exactly instead of text-searching their fragments
All checks were successful
ci/woodpecker/pr/changelog Pipeline was successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/unit-testing-elixir-1.19 Pipeline was successful
ci/woodpecker/pr/unit-testing-elixir-1.15 Pipeline was successful
86ddc9e771
A URL query (how clients resolve links) OR'ed the URL's fragments through
the text index, so "de" or "org" matched tens of thousands of accounts
that were then all trigram-ranked: 1-2 s per search on a 1M-user
instance. URLs now match by ap_id, uri, resolution and the profile URL's
name@host nickname only.
Author
Owner

@opencode-beta what do you think of this?

@opencode-beta what do you think of this?
First-time contributor

Blocked: base of this PR moved during processing — result withheld as stale.

Initial verification of this PR against the job's captured state passed (repo pleroma/pleroma, PR #7982, state open, base develop@124f4ece1d2f0786e3774d750cdd84440e744517, head fix/query-hotspots@86ddc9e7716a77e927b8a1baf7553309c7c9ec6c). Immediately before publishing, a live re-check found the captured base state had changed:

  • Base develop tip is now fa920037f16254e89d87252d7e4322b2b885e476 — the merge of companion PR #7981 ("Fix notification query plan for users with domain blocks") into develop.
  • Head is unchanged: fix/query-hotspots@86ddc9e7716a77e927b8a1baf7553309c7c9ec6c, PR still open, retargeting not detected.

Because the captured base SHA no longer matches live state, policy is to stop without publishing the substantive result. No review, branch, push, or other mutation was made by this job; the exact head-to-base diff on the recorded topology was reviewed locally and the outcome (plus this stale-base note) will need a fresh trigger on the updated state.

Marker: opencode-agent:job=1:generation=1

**Blocked: base of this PR moved during processing — result withheld as stale.** Initial verification of this PR against the job's captured state passed (repo `pleroma/pleroma`, PR #7982, state open, base `develop`@`124f4ece1d2f0786e3774d750cdd84440e744517`, head `fix/query-hotspots`@`86ddc9e7716a77e927b8a1baf7553309c7c9ec6c`). Immediately before publishing, a live re-check found the captured base state had changed: - **Base `develop` tip is now `fa920037f16254e89d87252d7e4322b2b885e476`** — the merge of companion PR #7981 ("Fix notification query plan for users with domain blocks") into develop. - Head is unchanged: `fix/query-hotspots`@`86ddc9e7716a77e927b8a1baf7553309c7c9ec6c`, PR still open, retargeting not detected. Because the captured base SHA no longer matches live state, policy is to stop without publishing the substantive result. No review, branch, push, or other mutation was made by this job; the exact head-to-base diff on the recorded topology was reviewed locally and the outcome (plus this stale-base note) will need a fresh trigger on the updated state. Marker: opencode-agent:job=1:generation=1
Author
Owner

@opencode-beta again, what do you think of this?

@opencode-beta again, what do you think of this?
First-time contributor

@lambadalambda Review of this PR as it stands now — short version: all four fixes look correct, well-scoped and well-tested; I'd be comfortable seeing this merged. This is the follow-up to the previously withheld review (the base had moved mid-job last time); the base has since settled on the #7981 merge and the analysis below is against the current state, which matches what I captured this run.

Verified live before publishing: repo pleroma/pleroma (id 13, not a fork), PR #7982 state open (captured and current: open), base develop@fa920037f16254e89d87252d7e4322b2b885e476 (captured = current; it is the tip of develop), head fix/query-hotspots@86ddc9e7716a77e927b8a1baf7553309c7c9ec6c (captured = current), title and body unchanged vs. my captured snapshot.

Per-commit verdicts (branch is exactly the 4 claimed commits; merge-base diff is 10 files, +172/−27 — 4 changelogs, stats.ex, user/search.ex, activity_pub.ex, 3 test files):

  1. Stats peers hourly (e0b5a22e3) — correct. The GenServer state starts as nil until handle_continue(:calculate_stats, _), and handle_info(:run_update, state) guards with is_map(state) before reading state.peers, recomputing otherwise — consistent with init. Both ticks self-reschedule, so the 1 min/1 h cadence is preserved, and force_update/0 still does the full recompute as the description claims. Nit: state.peers would raise KeyError if the state were ever a map without :peers; Map.get(state, :peers) would be more defensive, but every reachable state carries the key today.

  2. tag_reject anti-join (a8d0584d5) — semantics preserved, plan improved. NOT IN (subselect) → correlated NOT EXISTS probing the hashtags_objects pkey. parent_as(:object) is sound: the activities query names its object join as: :object (Activity.with_joined_object/2), and skip_preload still raises as before. The hashtag_ids == [] early-exit reproduces the old empty-subquery behavior and is pinned by the new "ignores tag_reject for tags nobody used" test. Cost is one extra small indexed lookup over hashtags per query build — negligible.

  3. Single hashtag = (8d69b1c4b) — behavior-identical special case ([id] -> ==, otherwise in); only the plan shape changes (walk (hashtag_id, object_id) newest-first and stop at limit, vs. walk the global object_id index and filter). The in-code comment documenting the PostgreSQL 15 observation is a good idea. Empty tag list still degrades to in [] → no rows, unchanged.

  4. URL searches exact (86ddc9e77) — deliberate behavior change, well covered by tests. fts_search short-circuits http* queries to where(query, false) so URL fragments stop trigram-matching tens of thousands of accounts; profile URLs resolve via the last path segment → name@host nickname, with an exit-catching wrapper around :idna.encode (which exits rather than raises) and tests for unusable URLs. Result ordering stays sensible (resolved > ap_id > uri > profile URL). The two updated tests correctly drop the com_user text-search artifact. Worth a line in release notes: URL searches are now exact-only — e.g. a status URL (.../statuses/123) no longer matches its author, which matches the stated intent.

The pre-existing issue flagged in the description (top-user URL matches bypassing block/invisible/unreachable filters via or_where in select_top_users) is confirmed present and untouched here — agreed it belongs in its own issue. Despite the branch being based on pre-#7981 develop (f53342775), git merge-tree shows it merges cleanly into current develop.

Caveats on this review: static analysis only — my environment has no Elixir/Erlang/Postgres, so I could not run the stats / user search / activity_pub / controller suites locally; each fix does come with its own tests, and no CI status is published on the head commit that I could read via the API. The query-plan/timing numbers in the description are taken as reported, not re-measured. No remote operations were performed or needed for this result: no branch pushed, no code changes committed, no review state set, nothing merged. If the branch moves or you want me to dig into any of the nits above, tag @opencode-beta again.

Marker: opencode-agent:job=2:generation=1

@lambadalambda Review of this PR as it stands now — short version: **all four fixes look correct, well-scoped and well-tested; I'd be comfortable seeing this merged.** This is the follow-up to the previously withheld review (the base had moved mid-job last time); the base has since settled on the #7981 merge and the analysis below is against the current state, which matches what I captured this run. **Verified live before publishing:** repo `pleroma/pleroma` (id 13, not a fork), PR #7982 state `open` (captured and current: open), base `develop`@`fa920037f16254e89d87252d7e4322b2b885e476` (captured = current; it is the tip of develop), head `fix/query-hotspots`@`86ddc9e7716a77e927b8a1baf7553309c7c9ec6c` (captured = current), title and body unchanged vs. my captured snapshot. **Per-commit verdicts** (branch is exactly the 4 claimed commits; merge-base diff is 10 files, +172/−27 — 4 changelogs, `stats.ex`, `user/search.ex`, `activity_pub.ex`, 3 test files): 1. **Stats peers hourly (`e0b5a22e3`)** — correct. The GenServer state starts as `nil` until `handle_continue(:calculate_stats, _)`, and `handle_info(:run_update, state)` guards with `is_map(state)` before reading `state.peers`, recomputing otherwise — consistent with `init`. Both ticks self-reschedule, so the 1 min/1 h cadence is preserved, and `force_update/0` still does the full recompute as the description claims. Nit: `state.peers` would raise `KeyError` if the state were ever a map without `:peers`; `Map.get(state, :peers)` would be more defensive, but every reachable state carries the key today. 2. **`tag_reject` anti-join (`a8d0584d5`)** — semantics preserved, plan improved. `NOT IN (subselect)` → correlated `NOT EXISTS` probing the `hashtags_objects` pkey. `parent_as(:object)` is sound: the activities query names its object join `as: :object` (`Activity.with_joined_object/2`), and `skip_preload` still raises as before. The `hashtag_ids == []` early-exit reproduces the old empty-subquery behavior and is pinned by the new "ignores tag_reject for tags nobody used" test. Cost is one extra small indexed lookup over `hashtags` per query build — negligible. 3. **Single hashtag `=` (`8d69b1c4b`)** — behavior-identical special case (`[id] -> ==`, otherwise `in`); only the plan shape changes (walk `(hashtag_id, object_id)` newest-first and stop at limit, vs. walk the global `object_id` index and filter). The in-code comment documenting the PostgreSQL 15 observation is a good idea. Empty tag list still degrades to `in []` → no rows, unchanged. 4. **URL searches exact (`86ddc9e77`)** — deliberate behavior change, well covered by tests. `fts_search` short-circuits `http*` queries to `where(query, false)` so URL fragments stop trigram-matching tens of thousands of accounts; profile URLs resolve via the last path segment → `name@host` nickname, with an exit-catching wrapper around `:idna.encode` (which exits rather than raises) and tests for unusable URLs. Result ordering stays sensible (resolved > ap_id > uri > profile URL). The two updated tests correctly drop the `com_user` text-search artifact. Worth a line in release notes: URL searches are now exact-only — e.g. a status URL (`.../statuses/123`) no longer matches its author, which matches the stated intent. The pre-existing issue flagged in the description (top-user URL matches bypassing block/invisible/unreachable filters via `or_where` in `select_top_users`) is confirmed present and untouched here — agreed it belongs in its own issue. Despite the branch being based on pre-#7981 develop (`f53342775`), `git merge-tree` shows it merges cleanly into current `develop`. **Caveats on this review:** static analysis only — my environment has no Elixir/Erlang/Postgres, so I could not run the stats / user search / activity_pub / controller suites locally; each fix does come with its own tests, and no CI status is published on the head commit that I could read via the API. The query-plan/timing numbers in the description are taken as reported, not re-measured. No remote operations were performed or needed for this result: no branch pushed, no code changes committed, no review state set, nothing merged. If the branch moves or you want me to dig into any of the nits above, tag @opencode-beta again. Marker: opencode-agent:job=2:generation=1
All checks were successful
ci/woodpecker/pr/changelog Pipeline was successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/unit-testing-elixir-1.19 Pipeline was successful
ci/woodpecker/pr/unit-testing-elixir-1.15 Pipeline was successful
This pull request has changes conflicting with the target branch.
  • lib/pleroma/stats.ex
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/query-hotspots:fix/query-hotspots
git switch fix/query-hotspots

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch develop
git merge --no-ff fix/query-hotspots
git switch fix/query-hotspots
git rebase develop
git switch develop
git merge --ff-only fix/query-hotspots
git switch fix/query-hotspots
git rebase develop
git switch develop
git merge --no-ff fix/query-hotspots
git switch develop
git merge --squash fix/query-hotspots
git switch develop
git merge --ff-only fix/query-hotspots
git switch develop
git merge fix/query-hotspots
git push origin develop
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
pleroma/pleroma!7982
No description provided.