After being down for a day, Pleroma runs insanely slow
My Pleroma instance was down for around one day, because I messed up the system it was running on. After reinstalling and importing a full db backup, the instance came back up, but it runs very slowly, with many errors (section of log below)
Dec 31 17:35:00 blobcat mix[7981]: Server: blob.cat:80 (http)
Dec 31 17:35:00 blobcat mix[7981]: Request: GET /api/v1/timelines/home?count=20&with_muted=true
Dec 31 17:35:00 blobcat mix[7981]: ** (exit) an exception was raised:
Dec 31 17:35:00 blobcat mix[7981]: ** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
Dec 31 17:35:00 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
Dec 31 17:35:00 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
Dec 31 17:35:00 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
Dec 31 17:35:00 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Dec 31 17:35:00 blobcat mix[7981]: (pleroma) lib/pleroma/pagination.ex:36: Pleroma.Pagination.fetch_paginated/4
Dec 31 17:35:00 blobcat mix[7981]: (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:1163: Pleroma.Web.ActivityPub.ActivityPub.fetch_activities/3
Dec 31 17:35:00 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:36: Pleroma.Web.MastodonAPI.TimelineController.home/2
Dec 31 17:35:00 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:5: Pleroma.Web.MastodonAPI.TimelineController.action/2
Dec 31 17:35:05 blobcat mix[7981]: 17:35:05.551 [info] Postgrex.Protocol (#PID<0.570.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13178.16> exited
Dec 31 17:35:06 blobcat mix[7981]: 17:35:06.243 [error] Postgrex.Protocol (#PID<0.572.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13554.16> timed out because it queued and checked out the connection for longer than 15000ms
Dec 31 17:35:06 blobcat mix[7981]: #PID<0.13554.16> was at location:
Dec 31 17:35:06 blobcat mix[7981]: :prim_inet.recv0/3
Dec 31 17:35:06 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:2837: Postgrex.Protocol.msg_recv/4
Dec 31 17:35:06 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:1879: Postgrex.Protocol.recv_bind/3
Dec 31 17:35:06 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:1771: Postgrex.Protocol.bind_execute/4
Dec 31 17:35:06 blobcat mix[7981]: (db_connection) lib/db_connection/holder.ex:293: DBConnection.Holder.holder_apply/4
Dec 31 17:35:06 blobcat mix[7981]: (db_connection) lib/db_connection.ex:1255: DBConnection.run_execute/5
Dec 31 17:35:06 blobcat mix[7981]: (db_connection) lib/db_connection.ex:1342: DBConnection.run/6
Dec 31 17:35:06 blobcat mix[7981]: (db_connection) lib/db_connection.ex:540: DBConnection.parsed_prepare_execute/5
Dec 31 17:35:06 blobcat mix[7981]: 17:35:06.244 request_id=FeWFMUWf8u6r2aMBH2xx [error] Internal server error: %DBConnection.ConnectionError{message: "tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)", severity: :error}
Dec 31 17:35:06 blobcat mix[7981]: 17:35:06.245 request_id=FeWFMUWf8u6r2aMBH2xx [info] Converted error DBConnection.ConnectionError to 500 response
Dec 31 17:35:06 blobcat mix[7981]: 17:35:06.250 [error] #PID<0.13554.16> running Pleroma.Web.Endpoint (connection #PID<0.13739.16>, stream id 1) terminated
Dec 31 17:35:06 blobcat mix[7981]: Server: blob.cat:80 (http)
Dec 31 17:35:06 blobcat mix[7981]: Request: GET /api/v1/favourites?count=20&with_muted=true
Dec 31 17:35:06 blobcat mix[7981]: ** (exit) an exception was raised:
Dec 31 17:35:06 blobcat mix[7981]: ** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
Dec 31 17:35:06 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
Dec 31 17:35:06 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
Dec 31 17:35:06 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
Dec 31 17:35:06 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Dec 31 17:35:06 blobcat mix[7981]: (pleroma) lib/pleroma/pagination.ex:36: Pleroma.Pagination.fetch_paginated/4
Dec 31 17:35:06 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/status_controller.ex:350: Pleroma.Web.MastodonAPI.StatusController.favourites/2
Dec 31 17:35:06 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/status_controller.ex:5: Pleroma.Web.MastodonAPI.StatusController.action/2
Dec 31 17:35:06 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/status_controller.ex:5: Pleroma.Web.MastodonAPI.StatusController.phoenix_controller_pipeline/2
Dec 31 17:35:09 blobcat mix[7981]: 17:35:09.254 [error] Postgrex.Protocol (#PID<0.568.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.13744.16> timed out because it queued and checked out the connection for longer than 15000ms
Dec 31 17:35:09 blobcat mix[7981]: #PID<0.13744.16> was at location:
Dec 31 17:35:09 blobcat mix[7981]: :prim_inet.recv0/3
Dec 31 17:35:09 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:2837: Postgrex.Protocol.msg_recv/4
Dec 31 17:35:09 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:1879: Postgrex.Protocol.recv_bind/3
Dec 31 17:35:09 blobcat mix[7981]: (postgrex) lib/postgrex/protocol.ex:1771: Postgrex.Protocol.bind_execute/4
Dec 31 17:35:09 blobcat mix[7981]: (db_connection) lib/db_connection/holder.ex:293: DBConnection.Holder.holder_apply/4
Dec 31 17:35:09 blobcat mix[7981]: (db_connection) lib/db_connection.ex:1255: DBConnection.run_execute/5
Dec 31 17:35:09 blobcat mix[7981]: (db_connection) lib/db_connection.ex:1342: DBConnection.run/6
Dec 31 17:35:09 blobcat mix[7981]: (db_connection) lib/db_connection.ex:596: DBConnection.execute/4
Dec 31 17:35:09 blobcat mix[7981]: 17:35:09.263 request_id=FeWFMf37oSyN__QBH27x [error] Internal server error: %DBConnection.ConnectionError{message: "tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)", severity: :error}
Dec 31 17:35:09 blobcat mix[7981]: 17:35:09.263 request_id=FeWFMf37oSyN__QBH27x [info] Converted error DBConnection.ConnectionError to 500 response
Dec 31 17:35:09 blobcat mix[7981]: 17:35:09.265 [error] #PID<0.13744.16> running Pleroma.Web.Endpoint (connection #PID<0.13746.16>, stream id 1) terminated
Dec 31 17:35:09 blobcat mix[7981]: Server: blob.cat:80 (http)
Dec 31 17:35:09 blobcat mix[7981]: Request: GET /api/v1/timelines/home?count=20&with_muted=true
Dec 31 17:35:09 blobcat mix[7981]: ** (exit) an exception was raised:
Dec 31 17:35:09 blobcat mix[7981]: ** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
Dec 31 17:35:09 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
Dec 31 17:35:09 blobcat mix[7981]: (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
Dec 31 17:35:09 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
Dec 31 17:35:09 blobcat mix[7981]: (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Dec 31 17:35:09 blobcat mix[7981]: (pleroma) lib/pleroma/pagination.ex:36: Pleroma.Pagination.fetch_paginated/4
Dec 31 17:35:09 blobcat mix[7981]: (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:1163: Pleroma.Web.ActivityPub.ActivityPub.fetch_activities/3
Dec 31 17:35:09 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:36: Pleroma.Web.MastodonAPI.TimelineController.home/2
Dec 31 17:35:09 blobcat mix[7981]: (pleroma) lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex:5: Pleroma.Web.MastodonAPI.TimelineController.action/2
Dec 31 17:35:11 blobcat mix[7981]: 17:35:11.557 [error] Error while fetching https://mastodon.technology/users/mdszy/statuses/102884953032695400: {:error, "Object has been deleted"}
Dec 31 17:35:11 blobcat mix[7981]: 17:35:11.891 [error] Error while fetching https://mastodon.technology/users/mdszy/statuses/102883223096313547: {:error, "Object has been deleted"}
Dec 31 17:35:12 blobcat mix[7981]: 17:35:12.430 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Dec 31 17:35:12 blobcat mix[7981]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1066ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
Dec 31 17:35:12 blobcat mix[7981]: (db_connection) lib/db_connection.ex:746: DBConnection.run/3
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Dec 31 17:35:12 blobcat mix[7981]: Last message: :poll
Dec 31 17:35:12 blobcat mix[7981]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [], name: Oban, node: "master_0c3bf0c8@127.0.0.1", poll_interval: 1000, prefix: "public", prune: {:maxlen, 1500}, prune_interval: 60000, prune_limit: 5000, queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5], repo: Pleroma.Repo, rescue_after: 60, rescue_interval: 60000, shutdown_grace_period: 15000, verbose: false}, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "gvtgamad", paused: false, poll_ref: #Reference<0.887427150.3013083137.6501>, queue: "federator_outgoing", rescue_ref: #Reference<0.887427150.3012820993.241060>, running: %{}, started_at: ~U[2019-12-31 17:34:36.984351Z]}
Dec 31 17:35:12 blobcat mix[7981]: 17:35:12.438 [error] GenServer Oban.Queue.ScheduledActivities.Producer terminating
Dec 31 17:35:12 blobcat mix[7981]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1037ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
Dec 31 17:35:12 blobcat mix[7981]: (db_connection) lib/db_connection.ex:746: DBConnection.run/3
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Dec 31 17:35:12 blobcat mix[7981]: Last message: :poll
Dec 31 17:35:12 blobcat mix[7981]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [], name: Oban, node: "master_0c3bf0c8@127.0.0.1", poll_interval: 1000, prefix: "public", prune: {:maxlen, 1500}, prune_interval: 60000, prune_limit: 5000, queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5], repo: Pleroma.Repo, rescue_after: 60, rescue_interval: 60000, shutdown_grace_period: 15000, verbose: false}, foreman: Oban.Queue.ScheduledActivities.Foreman, limit: 10, name: Oban.Queue.ScheduledActivities.Producer, nonce: "isjvthsm", paused: false, poll_ref: #Reference<0.887427150.3013083137.6517>, queue: "scheduled_activities", rescue_ref: #Reference<0.887427150.3012820993.242407>, running: %{}, started_at: ~U[2019-12-31 17:34:39.316096Z]}
Dec 31 17:35:12 blobcat mix[7981]: 17:35:12.438 [error] GenServer Oban.Queue.Transmogrifier.Producer terminating
Dec 31 17:35:12 blobcat mix[7981]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1036ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
Dec 31 17:35:12 blobcat mix[7981]: (db_connection) lib/db_connection.ex:746: DBConnection.run/3
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Dec 31 17:35:12 blobcat mix[7981]: Last message: :poll
Dec 31 17:35:12 blobcat mix[7981]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [], name: Oban, node: "master_0c3bf0c8@127.0.0.1", poll_interval: 1000, prefix: "public", prune: {:maxlen, 1500}, prune_interval: 60000, prune_limit: 5000, queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5], repo: Pleroma.Repo, rescue_after: 60, rescue_interval: 60000, shutdown_grace_period: 15000, verbose: false}, foreman: Oban.Queue.Transmogrifier.Foreman, limit: 20, name: Oban.Queue.Transmogrifier.Producer, nonce: "4ftis8vg", paused: false, poll_ref: #Reference<0.887427150.3013083137.6533>, queue: "transmogrifier", rescue_ref: #Reference<0.887427150.3012820993.242415>, running: %{}, started_at: ~U[2019-12-31 17:34:39.317478Z]}
Dec 31 17:35:12 blobcat mix[7981]: 17:35:12.438 [error] GenServer Oban.Queue.ActivityExpiration.Producer terminating
Dec 31 17:35:12 blobcat mix[7981]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1032ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
Dec 31 17:35:12 blobcat mix[7981]: (db_connection) lib/db_connection.ex:746: DBConnection.run/3
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Dec 31 17:35:12 blobcat mix[7981]: Last message: :poll
Dec 31 17:35:12 blobcat mix[7981]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [], name: Oban, node: "master_0c3bf0c8@127.0.0.1", poll_interval: 1000, prefix: "public", prune: {:maxlen, 1500}, prune_interval: 60000, prune_limit: 5000, queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5], repo: Pleroma.Repo, rescue_after: 60, rescue_interval: 60000, shutdown_grace_period: 15000, verbose: false}, foreman: Oban.Queue.ActivityExpiration.Foreman, limit: 10, name: Oban.Queue.ActivityExpiration.Producer, nonce: "pdvrvllr", paused: false, poll_ref: #Reference<0.887427150.3013083137.6549>, queue: "activity_expiration", rescue_ref: #Reference<0.887427150.3012820993.242423>, running: %{}, started_at: ~U[2019-12-31 17:34:39.317742Z]}
Dec 31 17:35:12 blobcat mix[7981]: 17:35:12.438 [error] GenServer Oban.Queue.WebPush.Producer terminating
Dec 31 17:35:12 blobcat mix[7981]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1018ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
Dec 31 17:35:12 blobcat mix[7981]: (db_connection) lib/db_connection.ex:746: DBConnection.run/3
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
Dec 31 17:35:12 blobcat mix[7981]: (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Edited by Mew Mew