My instance can no longer render timelines reliably
As of two hours ago, my instance can no longer render timelines properly. Initially the local timeline, and then home timelines. I get errors like:
Nov 19 16:15:33 coyote mix[770]: 16:15:33.500 [error] Postgrex.Protocol (#PID<0.1248.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0
.6467.0> timed out because it queued and checked out the connection for longer than 15000ms
Nov 19 16:15:33 coyote mix[770]: #PID<0.6467.0> was at location:
Nov 19 16:15:33 coyote mix[770]: :prim_inet.recv0/3
Nov 19 16:15:33 coyote mix[770]: (postgrex 0.15.9) lib/postgrex/protocol.ex:2972: Postgrex.Protocol.msg_recv/4
Nov 19 16:15:33 coyote mix[770]: (postgrex 0.15.9) lib/postgrex/protocol.ex:2014: Postgrex.Protocol.recv_bind/3
Nov 19 16:15:33 coyote mix[770]: (postgrex 0.15.9) lib/postgrex/protocol.ex:1869: Postgrex.Protocol.bind_execute_close/4
Nov 19 16:15:33 coyote mix[770]: (db_connection 2.4.0) lib/db_connection/holder.ex:325: DBConnection.Holder.holder_apply/4
Nov 19 16:15:33 coyote mix[770]: (db_connection 2.4.0) lib/db_connection.ex:1314: DBConnection.run_execute/5
Nov 19 16:15:33 coyote mix[770]: (db_connection 2.4.0) lib/db_connection.ex:1409: DBConnection.run/6
Nov 19 16:15:33 coyote mix[770]: (db_connection 2.4.0) lib/db_connection.ex:631: DBConnection.execute/4
Nov 19 16:15:33 coyote mix[770]: 16:15:33.504 request_id=FykY50IMKjZer2wAAFMh [error] Internal server error: %DBConnection.ConnectionError{message: "tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)", reason: :error, severity: :error}
Nov 19 16:15:33 coyote mix[770]: 16:15:33.505 [error] #PID<0.6467.0> running Pleroma.Web.Endpoint (connection #PID<0.6466.0>, stream id 1) terminated
Nov 19 16:15:33 coyote mix[770]: Server: tailswish.industries:80 (http)
Nov 19 16:15:33 coyote mix[770]: Request: GET /api/v1/timelines/home?with_muted=false&limit=20
Nov 19 16:15:33 coyote mix[770]: ** (exit) an exception was raised:
Nov 19 16:15:33 coyote mix[770]: ** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
Nov 19 16:15:33 coyote mix[770]: (ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:760: Ecto.Adapters.SQL.raise_sql_call_error/1
Nov 19 16:15:33 coyote mix[770]: (ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:693: Ecto.Adapters.SQL.execute/5
Nov 19 16:15:33 coyote mix[770]: (ecto 3.6.2) lib/ecto/repo/queryable.ex:224: Ecto.Repo.Queryable.execute/4
Nov 19 16:15:33 coyote mix[770]: (ecto 3.6.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
Nov 19 16:15:33 coyote mix[770]: (pleroma 2.4.4-6-g112aea8e-tailswish) lib/pleroma/pagination.ex:40: Pleroma.Pagination.fetch_paginated/4
Nov 19 16:15:33 coyote mix[770]: (pleroma 2.4.4-6-g112aea8e-tailswish) lib/pleroma/web/activity_pub/activity_pub.ex:484: Pleroma.Web.ActivityPub.ActivityPub.fetch_activities/3
Nov 19 16:15:33 coyote mix[770]: (pleroma 2.4.4-6-g112aea8e-tailswish) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:56: Pleroma.Web.MastodonAPI.TimelineController.home/2
Nov 19 16:15:33 coyote mix[770]: (pleroma 2.4.4-6-g112aea8e-tailswish) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:5: Pleroma.Web.MastodonAPI.TimelineController.action/2
I have tried running both a remote prune and a full vacuum but neither one seems to have made an improvement.
This doesn't seem to have been triggered by any change in configuration, starting on its own this afternoon when I wasn't anywhere near the system.
Edited by Azure