Pleroma crashed in prod (blue screen of death, epic fail)
Just a few minutes ago I couldn't connect to my server. Out of nowhere it failed hard.
I checked journalctl -u pleroma
and found this:
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.169 [info] GET /emoji/2694.svg
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.178 [info] GET /emoji/1f6e1.svg
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.192 [info] Sent 200 in 23ms
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.201 [info] GET /emoji/2714.svg
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.212 [info] Sent 200 in 33ms
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.220 request_id=Fl2ad5pCrPHjPsQCRRPD [info] Chunked 206 in 72ms
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.224 [info] Sent 200 in 23ms
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.362 [info] GET /api/v1/lists
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.533 request_id=Fl2ad5j_R1-MnUACRROj [info] Chunked 200 in 407ms
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.833 [info] GET /proxy/l7Angrj5XJBWiqGnP8yu8C23qvs/aHR0cHM6Ly90aGVwb3N0bWlsbGVubmlhbC5jb20vY29udGVudC9pbWFnZXMvMjAyMS8wMS9VbnRpdGxlZC1kZXNpZ24tLS0yMDIxLTAxLTI1VDE1NTczMi44MDgucG5n/Untitled-design---2021-01-25T155732.808.png
Jan 25 23:34:40 gleasonator mix[1850101]: 23:34:40.869 request_id=Fl2ad8MmB7odYxUCRRPj [info] Chunked 206 in 35ms
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.118 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3390ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.121 [error] Elixir.Pleroma.Web.MastodonAPI.StatusView failed to render {Pleroma.Web.MastodonAPI.StatusView, "show.json"}
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 3389ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.exists?/3
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:242: Pleroma.Web.MastodonAPI.StatusView.render/2
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:6: Pleroma.Web.MastodonAPI.StatusView.safe_render/3
Jan 25 23:34:42 gleasonator mix[1850101]: (elixir 1.10.4) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.122 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3389ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.123 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3387ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.128 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3379ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.132 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3363ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.132 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3355ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.133 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3346ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.134 request_id=Fl2adcxbuuOs8LEExVBB [info] Sent 200 in 9737ms
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.134 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3330ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.135 request_id=Fl2adghTWH1Um9sDUJYi [info] Sent 200 in 8730ms
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.148 [warn] Instance.get_or_update_favicon("toot.cafe") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3322ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.150 request_id=Fl2ad6cDyqsGoQEExVlh [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1756ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.151 request_id=Fl2ad6cDyqsGoQEExVlh [info] Sent 500 in 1788ms
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.155 [error] #PID<0.14144.1165> running Pleroma.Web.Endpoint (connection #PID<0.6621.1165>, stream id 1) terminated
Jan 25 23:34:42 gleasonator mix[1850101]: Server: gleasonator.com:80 (http)
Jan 25 23:34:42 gleasonator mix[1850101]: Request: GET /api/v1/lists
Jan 25 23:34:42 gleasonator mix[1850101]: ** (exit) an exception was raised:
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1756ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:112: Ecto.Repo.Queryable.one/3
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/plugs/o_auth_plug.ex:57: Pleroma.Web.Plugs.OAuthPlug.fetch_user_and_token/1
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/plugs/o_auth_plug.ex:25: Pleroma.Web.Plugs.OAuthPlug.call/2
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) Pleroma.Web.Router.authenticate/2
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.155 [error] GenServer Oban.Queue.Transmogrifier.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2657ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584124, foreman: Oban.Queue.Transmogrifier.Foreman, limit: 20, name: Oban.Queue.Transmogrifier.Producer, nonce: "hb0p09ct", paused: false, poll_interval: 1000, queue: "transmogrifier", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.634040Z], timer: #Reference<0.2954531850.265814018.84107>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.168 [error] GenServer Oban.Queue.NewUsersDigest.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2656ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584123, foreman: Oban.Queue.NewUsersDigest.Foreman, limit: 1, name: Oban.Queue.NewUsersDigest.Producer, nonce: "sqrntlhr", paused: false, poll_interval: 1000, queue: "new_users_digest", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.634533Z], timer: #Reference<0.2954531850.265814018.84132>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.170 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 3303ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.184 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2793ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584252, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "sjhbj051", paused: false, poll_interval: 1000, queue: "federator_outgoing", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.634372Z], timer: #Reference<0.2954531849.3982753795.92350>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.185 [error] GenServer Oban.Queue.Backup.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2631ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584098, foreman: Oban.Queue.Backup.Foreman, limit: 1, name: Oban.Queue.Backup.Producer, nonce: "sbb0bsn1", paused: false, poll_interval: 1000, queue: "backup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.575643Z], timer: #Reference<0.2954531849.3982753795.92619>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.185 [error] GenServer Pleroma.Stats terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 3301ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:120: Ecto.Repo.Queryable.one!/3
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:84: Pleroma.Stats.calculate_stat_data/0
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:129: Pleroma.Stats.handle_info/2
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :run_update
Jan 25 23:34:42 gleasonator mix[1850101]: State: %{peers: ["cypher.social", "friendica.a-zwenkau.de", "tube.bootlicker.party", "pl.uwu.is", "schleuss.online", "mastodon.kaiserflur.de", "chilli.social", "ikbenpiraat.nl", "social.gnlk.cc", "mast.yellowfinch.cc", "glasgow.social", "theinternet.lol", "mastodon.nogods.be", "fla.red", "ap.umedasp.co.jp", "social.jw.be", "social.trans.town", "mastodon.partecipa.digital", "mindfire.io", "brfint.com", "fedi.lynnesbian.space", "misskey.imlostindeep.space", "pub.mayel.space", "utopia.cool", "hub.bierzilla.fr", "zee.li", "lily.network", "djsumdog.com", "sua.anarkis.net", "social.fab-l3.org", "imastodon.org", "dragon.style", "pyrosome.social", "mastodon.nzoss.nz", "jepsen.io", "pleroma.prolatio.xyz", "letsalllovela.in", "lagemeet.duckdns.org", "social.mafunke.de", "elephant.crime.group", "friends.grishka.me", "testing.muhroads.net", "nazidon-social.10minutepleroma.com", "thepicketts.online", "zerojanvier.fr", "tec.wtf", "masto.davewinter.info", "tmo.pizza", "social.port0.xyz", ...], stats: %{domain_count: 5418, status_count: 46415, user_count: 246}}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.186 [error] GenServer Oban.Queue.Mailer.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2655ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584122, foreman: Oban.Queue.Mailer.Foreman, limit: 10, name: Oban.Queue.Mailer.Producer, nonce: "894876js", paused: false, poll_interval: 1000, queue: "mailer", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633740Z], timer: #Reference<0.2954531850.265814018.84157>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.186 [error] GenServer Oban.Queue.TokenExpiration.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2654ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584121, foreman: Oban.Queue.TokenExpiration.Foreman, limit: 5, name: Oban.Queue.TokenExpiration.Producer, nonce: "takpso5e", paused: false, poll_interval: 1000, queue: "token_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633952Z], timer: #Reference<0.2954531850.265814018.84182>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.187 [error] GenServer Oban.Queue.Background.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2651ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584101, foreman: Oban.Queue.Background.Foreman, limit: 5, name: Oban.Queue.Background.Producer, nonce: "ub7898mr", paused: false, poll_interval: 1000, queue: "background", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.634609Z], timer: #Reference<0.2954531850.265814018.84289>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.187 [error] GenServer Oban.Queue.IngestionQueue.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2633ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584099, foreman: Oban.Queue.IngestionQueue.Foreman, limit: 50, name: Oban.Queue.IngestionQueue.Producer, nonce: "5ldphfge", paused: false, poll_interval: 1000, queue: "ingestion_queue", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633838Z], timer: #Reference<0.2954531850.265814018.84317>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.200 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 2974ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.200 [error] GenServer Oban.Queue.ScheduledActivities.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2659ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584126, foreman: Oban.Queue.ScheduledActivities.Foreman, limit: 10, name: Oban.Queue.ScheduledActivities.Producer, nonce: "jes97cdl", paused: false, poll_interval: 1000, queue: "scheduled_activities", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.634127Z], timer: #Reference<0.2954531850.265814018.84082>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.200 request_id=Fl2adseD56QfgfYDUJvi [info] Sent 200 in 5594ms
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.201 [error] GenServer Oban.Queue.RemoteFetcher.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2783ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584244, foreman: Oban.Queue.RemoteFetcher.Foreman, limit: 2, name: Oban.Queue.RemoteFetcher.Producer, nonce: "mbfnev7g", paused: false, poll_interval: 1000, queue: "remote_fetcher", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.575535Z], timer: #Reference<0.2954531850.948174849.196>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.202 [error] GenServer Oban.Queue.MuteExpire.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2769ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584226, foreman: Oban.Queue.MuteExpire.Foreman, limit: 5, name: Oban.Queue.MuteExpire.Producer, nonce: "vc1ntqld", paused: false, poll_interval: 1000, queue: "mute_expire", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633101Z], timer: #Reference<0.2954531850.948174849.323>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.217 [error] GenServer Oban.Queue.ActivityExpiration.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2727ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584181, foreman: Oban.Queue.ActivityExpiration.Foreman, limit: 10, name: Oban.Queue.ActivityExpiration.Producer, nonce: "g9m5k010", paused: false, poll_interval: 1000, queue: "activity_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633200Z], timer: #Reference<0.2954531850.948174849.455>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.218 [error] GenServer Oban.Queue.WebPush.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2776ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584237, foreman: Oban.Queue.WebPush.Foreman, limit: 50, name: Oban.Queue.WebPush.Producer, nonce: "nlrovp6d", paused: false, poll_interval: 1000, queue: "web_push", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.575609Z], timer: #Reference<0.2954531850.265814018.83939>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.218 [error] GenServer Oban.Queue.AttachmentsCleanup.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2758ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584202, foreman: Oban.Queue.AttachmentsCleanup.Foreman, limit: 5, name: Oban.Queue.AttachmentsCleanup.Producer, nonce: "g734mc0f", paused: false, poll_interval: 1000, queue: "attachments_cleanup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 09:27:31.633577Z], timer: #Reference<0.2954531850.265814018.83978>}
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.238 [error] GenServer Oban.Queue.FederatorIncoming.Producer terminating
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2728ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:42 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:42 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:42 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: -573184584005, foreman: Oban.Queue.FederatorIncoming.Foreman, limit: 50, name: Oban.Queue.FederatorIncoming.Producer, nonce: "499jpp4n", paused: false, poll_interval: 1000, queue: "federator_incoming", reset_timer: nil, running: %{#Reference<0.2954531849.3982753795.92872> => {%Oban.Queue.Executor{conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, duration: 0, error: nil, job: %Oban.Job{__meta__: #Ecto.Schema.Metadata<:loaded, "public", "oban_jobs">, args: %{"op" => "incoming_ap_doc", "params" => %{"@context" => ["https://www.w3.org/ns/activitystreams", %{"atomUri" => "ostatus:atomUri", "conversation" => "ostatus:conversation", "inReplyToAtomUri" => "ostatus:inReplyToAtomUri", "ostatus" => "http://ostatus.org#", "sensitive" => "as:sensitive", "toot" => "http://joinmastodon.org/ns#", "votersCount" => "toot:votersCount"}], "actor" => "https://paypig.org/users/SpecialLittleGuy", "cc" => ["https://paypig.org/users/SpecialLittleGuy/followers", "https://paypig.org/users/CrippledJesus", "https://paypig.org/users/karlwatp"], "id" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097/activity", "object" => %{"atomUri" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097", "attachment" => [], "attributedTo" => "https://paypig.org/users/SpecialLittleGuy", "cc" => ["https://paypig.org/users/SpecialLittleGuy/followers", "https://paypig.org/users/CrippledJesus", "https://paypig.org/users/karlwatp"], "content" => "<p><span class=\"h-card\"><a href=\"https://paypig.org/@karlwatp\" class=\"u-url mention\">@<span>karlwatp</span></a></span> <span class=\"h-card\"><a href=\"https://paypig.org/@CrippledJesus\" class=\"u-url mention\">@<span>CrippledJesus</span></a></span> Pretty sure you said you liked them on Shitty Song of the Week. I only remember because it was the first time I'd ever heard someone say the words "I like Five Finger Death Punch".</p>", "contentMap" => %{"en" => "<p><span class=\"h-card\"><a href=\"https://paypig.org/@karlwatp\" class=\"u-url mention\">@<span>karlwatp</span></a></span> <span class=\"h-card\"><a href=\"https://paypig.org/@CrippledJesus\" class=\"u-url mention\">@<span>CrippledJesus</span></a></span> Pretty sure you said you liked them on Shitty Song of the Week. I only remember because it was the first time I'd ever heard someone say the words "I like Five Finger Death Punch".</p>"}, "conversation" => "tag:paypig.org,2021-01-25:objectId=179976:objectType=Conversation", "id" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097", "inReplyTo" => "https://paypig.org/users/karlwatp/statuses/105618731238879832", "inReplyToAtomUri" => "https://paypig.org/users/karlwatp/statuses/105618731238879832", "published" => "2021-01-25T22:34:35Z", "replies" => %{"first" => %{"items" => [], "next" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097/replies?only_other_accounts=true&page=true", "partOf" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097/replies", "type" => "CollectionPage"}, "id" => "https://paypig.org/users/SpecialLittleGuy/statuses/105618740048797097/replies", "type" => "Collection"}, "sensitive" => false, "summary" => nil, "tag" => [%{"href" => "https://paypig.org/users/karlwatp", "name" => "@karlwatp", "type" => "Mention"}, %{"href" => "https://paypig.org/users/CrippledJesus", "name" => "@CrippledJesus", "type" => "Mention"}], "to" => ["https://www.w3.org/ns/activitystreams#Public"], "type" => "Note", "url" => "https://paypig.org/@SpecialLittleGuy/105618740048797097"}, "published" => "2021-01-25T22:34:35Z", "signature" => %{"created" => "2021-01-25T22:34:35Z", "creator" => "https://paypig.org/users/SpecialLittleGuy#main-key", "signatureValue" => "KjuH1864v1MbdOfI/GrQNGkcAe5FW1AXNF+ED3jq/m5pSxvtlncKf+QGyYtUVn2e0ZemXgKsBIcuqVn97wUpOIDP8ueHSMmYQlJTCIMAmmpOi/mTvKp3gE8JzSd1JRRK6gGhxukI3zQ2KSbx1VPLwTLjDqf339NZsGjIS9IcVexx16vBm4KGQ+1KSzh3cTYOfucw6jIN8goXCIrBAX1zSmUp1Ep6953RG9j+04yEHQ8vvrQGFzn+zwrtcrA8WR0+iwxo/KZK1rTxSc2EmmtrjshlbFhWK+9pDaswe+oIeSY7IgQLE5TpBiKaVsmRcrGbvrJkx2Rd+RN5EyI5fH8+Jg==", "type" => "RsaSignature2017"}, "to" => ["https://www.w3.org/ns/activitystreams#Public"], "type" => "Create"}}, attempt: 1, attempted_at: ~U[2021-01-25 22:34:38.435366Z], attempted_by: ["gleasonator", "federator_incoming", "499jpp4n"], completed_at: nil, discarded_at: nil, errors: [], id: 27385778, inserted_at: ~U[2021-01-25 22:34:38.430887Z], max_attempts: 5, priority: 0, queue: "federator_incoming", scheduled_at: ~U[2021-01-25 22:34:38.430887Z], state: "executing", tags: [], unique: nil, unsaved_error: nil, worker: "Pleroma.Workers.ReceiverWorker"}, kind: :error, meta: %{args: %{"op" => "incoming_ap_doc", "params" => %{"@context" => ["https://www.w3.org/ns/activitystreams", %{"atomUri" => "ostatus:atomUri", "conversation" => "ostatus:conversation", "inReplyToAtomUri" => "ostatus:inReplyToAtomUri", "ostatus" => "http://ostatus.org#", "sensitive" => "as:sensitive", "toot" => "http://joinmastodon.org/ns#", "votersCount" => "toot:votersCount"}], "actor" => "https://paypig.org/users/SpecialLittleGuy", "cc" => ["https://paypig.org/users/SpecialLittleGuy/f (truncated)
Jan 25 23:34:42 gleasonator mix[1850101]: 23:34:42.239 request_id=Fl2adpdnL69L5GUExVSh [error] Elixir.Pleroma.Web.MastodonAPI.NotificationView failed to render {Pleroma.Web.MastodonAPI.NotificationView, "show.json"}
Jan 25 23:34:42 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 3311ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:42 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:42 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:42 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:42 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:42 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.exists?/3
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:242: Pleroma.Web.MastodonAPI.StatusView.render/2
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/notification_view.ex:154: Pleroma.Web.MastodonAPI.NotificationView.put_status/4
Jan 25 23:34:42 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/notification_view.ex:6: Pleroma.Web.MastodonAPI.NotificationView.safe_render/3
Jan 25 23:34:43 gleasonator mix[1850101]: 23:34:43.369 [info] GET /api/v1/lists
Jan 25 23:34:43 gleasonator mix[1850101]: 23:34:43.538 [info] POST /inbox
Jan 25 23:34:43 gleasonator mix[1850101]: 23:34:43.707 request_id=Fl2aeGRfGWvE3rICRRRD [error] Could not decode user at fetch https://relay.civiq.social/actor, {:content_type, "application/json; charset=utf-8"}
Jan 25 23:34:43 gleasonator mix[1850101]: 23:34:43.878 request_id=Fl2aeGRfGWvE3rICRRRD [error] Could not decode user at fetch https://relay.civiq.social/actor, {:content_type, "application/json; charset=utf-8"}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.125 [error] Elixir.Pleroma.Web.MastodonAPI.SearchController search error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1996ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.126 request_id=Fl2aeGRfGWvE3rICRRRD [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 240ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.127 [error] Elixir.Pleroma.Web.MastodonAPI.StatusView failed to render {Pleroma.Web.MastodonAPI.StatusView, "show.json"}
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1998ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:112: Ecto.Repo.Queryable.one/3
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:225: Pleroma.Web.MastodonAPI.StatusView.render/2
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:6: Pleroma.Web.MastodonAPI.StatusView.safe_render/3
Jan 25 23:34:44 gleasonator mix[1850101]: (elixir 1.10.4) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.133 [warn] Instance.get_or_update_favicon("gab.com") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1933ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.133 request_id=Fl2adpdnL69L5GUExVSh [warn] Instance.get_or_update_favicon("freespeechextremist.com") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1858ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.141 [error] GenServer Oban.Queue.RemoteFetcher.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.RemoteFetcher.Foreman, limit: 2, name: Oban.Queue.RemoteFetcher.Producer, nonce: "38khd2u4", paused: false, poll_interval: 1000, queue: "remote_fetcher", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.218479Z], timer: #Reference<0.2954531849.3982753795.94760>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.172 [error] GenServer Pleroma.Stats terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1918ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:69: Pleroma.Stats.calculate_stat_data/0
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:107: Pleroma.Stats.handle_continue/2
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:431: :gen_server.loop/7
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: {:continue, :calculate_stats}
Jan 25 23:34:44 gleasonator mix[1850101]: State: nil
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.172 request_id=Fl2aeFpD92pgvo4DUKFC [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 749ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.179 [error] #PID<0.26153.1164> running Pleroma.Web.Endpoint (connection #PID<0.10410.1165>, stream id 1) terminated
Jan 25 23:34:44 gleasonator mix[1850101]: Server: gleasonator.com:80 (http)
Jan 25 23:34:44 gleasonator mix[1850101]: Request: POST /inbox
Jan 25 23:34:44 gleasonator mix[1850101]: ** (exit) an exception was raised:
Jan 25 23:34:44 gleasonator mix[1850101]: ** (FunctionClauseError) no function clause matching in Plug.Conn.resp/3
Jan 25 23:34:44 gleasonator mix[1850101]: (plug 1.10.4) lib/plug/conn.ex:580: Plug.Conn.resp(%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{digest: "SHA-256=JZVXUPCyknmeqeeKcGM9E/2rqbODAqOXC3xzxvCYSP4=", kind: :error, layout: false, locale: "en", reason: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 240ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}, remote_ip_found: true, stack: [{Ecto.Adapters.SQL, :raise_sql_call_error, 1, [file: 'lib/ecto/adapters/sql.ex', line: 593]}, {Ecto.Adapters.SQL, :execute, 5, [file: 'lib/ecto/adapters/sql.ex', line: 526]}, {Ecto.Repo.Queryable, :execute, 4, [file: 'lib/ecto/repo/queryable.ex', line: 192]}, {Ecto.Repo.Queryable, :all, 3, [file: 'lib/ecto/repo/queryable.ex', line: 17]}, {Ecto.Repo.Queryable, :one, 3, [file: 'lib/ecto/repo/queryable.ex', line: 112]}, {Pleroma.Instances.Instance, :set_reachable, 1, [file: 'lib/pleroma/instances/instance.ex', line: 84]}, {Pleroma.Web.ActivityPub.ActivityPubController, :set_requester_reachable, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 515]}, {Pleroma.Web.ActivityPub.ActivityPubController, :phoenix_controller_pipeline, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 5]}], status: 500, token: nil, user: %Pleroma.User{password_confirmation: nil, keys: nil, muted_users: #Ecto.Association.NotLoaded<association :muted_users is not loaded>, name: "ActivityRelay", uri: "https://relay.civiq.social/actor", mutes: [], confirmation_token: nil, hide_follows: false, fields: [], password_reset_pending: false, subscribee_users: #Ecto.Association.NotLoaded<association :subscribee_users is not loaded>, muted_notifications: [], reblog_mutee_mutes: #Ecto.Association.NotLoaded<association :reblog_mutee_mutes is not loaded>, avatar: nil, note_count: 1212373, updated_at: ~N[2020-06-10 02:15:31], registrations: #Ecto.Association.NotLoaded<association :registrations is not loaded>, nickname: "relay@relay.civiq.social", background: %{}, following_address: "https://relay.civiq.social/following", confirmation_pending: true, follower_count: 0, multi_factor_authentication_settings: %Pleroma.MFA.Settings{backup_codes: [], enabled: false, totp: nil}, muter_users: #Ecto.Association.NotLoaded<association :muter_users is not loaded>, pinned_activities: [], following_count: 0, search_type: nil, hide_followers: false, default_scope: "public", invisible: false, follower_address: "https://relay.civiq.social/followers", is_locked: false, inserted_at: ~N[2020-01-08 01:41:14], shared_inbox: "https://relay.civiq.social/inbox", email_notifications: %{"digest" => false}, muted_reblogs: [], tags: [], reblog_muter_mutes: #Ecto.Association.NotLoaded<association :reblog_muter_mutes is not loaded>, ...}, valid_signature: true}, before_send: [#Function<0.119468924/1 in Pleroma.Web.Endpoint.PipelineInstrumenter.call/2>, #Function<1.132129851/1 in Plug.Logger.call/2>, #Function<0.60895335/1 in Plug.Telemetry.call/2>], body_params: %{"@context" => "https://www.w3.org/ns/activitystreams", "actor" => "https://relay.civiq.social/actor", "id" => "https://relay.civiq.social/activities/4e15c2e8-9316-4f5d-82eb-44afe98e321c", "object" => "https://mstdn.foxfam.club/users/epochtimes/statuses/105618296671487138", "to" => ["https://relay.civiq.social/followers"], "type" => "Announce"}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "gleasonator.com", method: "POST", owner: #PID<0.26153.1164>, params: %{"@context" => "https://www.w3.org/ns/activitystreams", "actor" => "https://relay.civiq.social/actor", "id" => "https://relay.civiq.social/activities/4e15c2e8-9316-4f5d-82eb-44afe98e321c", "object" => "https://mstdn.foxfam.club/users/epochtimes/statuses/105618296671487138", "to" => ["https://relay.civiq.social/followers"], "type" => "Announce"}, path_info: ["inbox"], path_params: %{}, port: 80, private: %{Pleroma.Web.Router => {[], %{}}, :phoenix_endpoint => Pleroma.Web.Endpoint, :phoenix_format => "activity+json", :phoenix_layout => false, :phoenix_root_layout => false, :phoenix_router => Pleroma.Web.Router, :phoenix_template => "500.activity+json", :phoenix_view => Pleroma.Web.ErrorView, :plug_session_fetch => #Function<1.105793137/1 in Plug.Session.fetch_session/1>, :skipped_plugs => [Pleroma.Web.Plugs.OAuthScopesPlug]}, query_params: %{}, query_string: "", remote_ip: {45, 32, 204, 240}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "*/*"}, {"accept-encoding", "gzip"}, {"cdn-loop", "cloudflare"}, {"cf-connecting-ip", "45.32.204.240"}, {"cf-ipcountry", "US"}, {"cf-ray", "6175768e4affef5d-NRT"}, {"cf-request-id", "07dd486cee0000ef5d832a5000000001"}, {"cf-visitor", "{\"scheme\":\"https\"}"}, {"connection", "upgrade"}, {"content-length", "334"}, {"content-type", "application/activity+json"}, {"host", "gleasonator.com"}, {"signature", "keyId=\"https://relay.civiq.social/actor#main-key\",algorithm=\"rsa-sha256\",headers=\"(request-target) content-length content-type user-agent\",signature=\"OJGBi2v7KrT5Or/vWjGCojgrqx9iuacrr70GCc/aRMkd2k98YGK/AFSjue2cvNaOQ5BtocChrU/X7sWkKCHVdGJoPPj1tC5JLAf9cNzBM7RVtQygm06X2EET/wGKEo5ogqnDjDxQqFaLQlA473zMpCJq+xHXQtLygCyYLVRvG6tr62va5MU3p0CEem6ShBIVNwkySX++909PnOqI/tloLIxoxj32GRKx22WEgysjyj+V6Pmlqm2YmHQtGci1+9an7Ik6LHoEEIdPIl4bTLzUGe/MrjJrFObN9yS8BtryGyI165vEIBC9UkOmkxYopSud0BuMDnA/KM10NLfpDy8roYdRMd0LzDECIhJAbHNdUR9Zjzvo53l6/Ujn0milJbG16+9jbEJbQIBggMMvKHs2tO2NjM3jYP0ZTgTfQdBNFjAGy/pJpTALjzV8w4Rds4BYij9/6UlE8EO1nb2FRWwTkjPgbQhYIVF1NSalDgCoLjDQlq1Bsg7nKpKz3LNJ8/bfKho5vFIgxMzrCXJaL3uCF+8sN+QKK7r+o1vML5PKzEyMq2FCfwHT5kJwV+snSWhwUy/oRc0UrnaIwtadKGSGjOMRJF7JOOqErfmbj8noVLQAybuoR0CsnC87BrrG9Y1ffSYeaQPgQOe+n7in66bXNFs4GwRdmx9xlt4/PtS6XyM=\""}, {"user-agent", "ActivityRelay"}, {"x-forwarded-for", "45.32.204.240, 45.32.204.240"}, {"x-forwarded-proto", "https"}, {"(request-target)", "post /inbox"}, {"digest", "SHA-256=JZVXUPCyknmeqeeKcGM9E/2rqbODAqOXC3xzxvCYSP4="}], request_path: "/inbox", resp_body: nil, resp_cookies: %{}, resp_headers: [{"content-type", "application/activity+json; charset=utf-8"}, {"cache-control", "max-age=0, private, must-revalidate"}, {"access-control-allow-origin", "*"}, {"access-control-expose-headers", "Link,X-RateLimit-Reset,X-RateLimit-Limit,X-RateLimit-Remaining,X-Request-Id,Idempotency-Key"}, {"access-control-allow-credentials", "true"}, {"x-xss-protection", "1; mode=block"}, {"x-permitted-cross-domain-policies", "none"}, {"x-frame-options", "DENY"}, {"x-content-type-options", "nosniff"}, {"referrer-policy", "same-origin"}, {"x-download-options", "noopen"}, {"content-security-policy", "upgrade-insecure-requests;script-src 'self';connect-src 'self' blob: https://gleasonator.com wss://gleasonator.com;media-src 'self' https://s3.amazonaws.com https://media.gleasonator.com;img-src 'self' data: blob: https://s3.amazonaws.com https://media.gleasonator.com;default-src 'none';base-uri 'self';frame-ancestors 'none';style-src 'self' 'unsafe-inline';font-src 'self';manifest-src 'self';"}, {"strict-transport-security", "max-age=31536000; includeSubDomains"}, {"expect-ct", "enforce, max-age=2592000"}, {"x-request-id", "Fl2aeGRfGWvE3rICRRRD"}], scheme: :http, script_name: [], secret_key_base: :..., state: :unset, status: 500}, 500, %{errors: %{detail: "Internal server error"}})
Jan 25 23:34:44 gleasonator mix[1850101]: (plug 1.10.4) lib/plug/conn.ex:559: Plug.Conn.send_resp/3
Jan 25 23:34:44 gleasonator mix[1850101]: (phoenix 1.5.6) lib/phoenix/endpoint/render_errors.ex:78: P (truncated)
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.196 request_id=Fl2aeFpD92pgvo4DUKFC [info] Sent 500 in 826ms
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.197 request_id=Fl2adwkaNYLEZ2EExVdh [info] Sent 200 in 6484ms
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.228 [error] #PID<0.30791.1164> running Pleroma.Web.Endpoint (connection #PID<0.27583.1164>, stream id 1) terminated
Jan 25 23:34:44 gleasonator mix[1850101]: Server: gleasonator.com:80 (http)
Jan 25 23:34:44 gleasonator mix[1850101]: Request: GET /api/v1/lists
Jan 25 23:34:44 gleasonator mix[1850101]: ** (exit) an exception was raised:
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 749ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:44 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:112: Ecto.Repo.Queryable.one/3
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/plugs/o_auth_plug.ex:57: Pleroma.Web.Plugs.OAuthPlug.fetch_user_and_token/1
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/plugs/o_auth_plug.ex:25: Pleroma.Web.Plugs.OAuthPlug.call/2
Jan 25 23:34:44 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) Pleroma.Web.Router.authenticate/2
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.231 [error] GenServer Oban.Queue.Mailer.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Mailer.Foreman, limit: 10, name: Oban.Queue.Mailer.Producer, nonce: "jm28su67", paused: false, poll_interval: 1000, queue: "mailer", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.186485Z], timer: #Reference<0.2954531849.3982753795.94767>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.245 [error] GenServer Oban.Queue.Background.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Background.Foreman, limit: 5, name: Oban.Queue.Background.Producer, nonce: "fuh2006d", paused: false, poll_interval: 1000, queue: "background", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.187388Z], timer: #Reference<0.2954531849.3982753795.94769>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.246 [error] GenServer Oban.Queue.ActivityExpiration.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 898ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ActivityExpiration.Foreman, limit: 10, name: Oban.Queue.ActivityExpiration.Producer, nonce: "6m41nbcs", paused: false, poll_interval: 1000, queue: "activity_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.219469Z], timer: #Reference<0.2954531850.948174849.2641>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.252 [error] GenServer Oban.Queue.AttachmentsCleanup.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 898ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.AttachmentsCleanup.Foreman, limit: 5, name: Oban.Queue.AttachmentsCleanup.Producer, nonce: "8i73f4cl", paused: false, poll_interval: 1000, queue: "attachments_cleanup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.219573Z], timer: #Reference<0.2954531850.948174849.2644>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.253 [error] GenServer Oban.Queue.TokenExpiration.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.TokenExpiration.Foreman, limit: 5, name: Oban.Queue.TokenExpiration.Producer, nonce: "m4v71c96", paused: false, poll_interval: 1000, queue: "token_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.186981Z], timer: #Reference<0.2954531849.3982753795.94768>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.254 [error] GenServer Oban.Queue.ScheduledActivities.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 886ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ScheduledActivities.Foreman, limit: 10, name: Oban.Queue.ScheduledActivities.Producer, nonce: "g4csumk6", paused: false, poll_interval: 1000, queue: "scheduled_activities", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.219734Z], timer: #Reference<0.2954531850.948174849.2647>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.254 [error] GenServer Oban.Queue.FederatorIncoming.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 880ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorIncoming.Foreman, limit: 50, name: Oban.Queue.FederatorIncoming.Producer, nonce: "cstf70go", paused: false, poll_interval: 1000, queue: "federator_incoming", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.238263Z], timer: #Reference<0.2954531850.948174849.2651>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.255 [error] GenServer Oban.Queue.Transmogrifier.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 932ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Transmogrifier.Foreman, limit: 20, name: Oban.Queue.Transmogrifier.Producer, nonce: "5im2jsrc", paused: false, poll_interval: 1000, queue: "transmogrifier", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.168436Z], timer: #Reference<0.2954531850.948174849.2615>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.255 [error] GenServer Oban.Queue.NewUsersDigest.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 932ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.NewUsersDigest.Foreman, limit: 1, name: Oban.Queue.NewUsersDigest.Producer, nonce: "l6qbu4hi", paused: false, poll_interval: 1000, queue: "new_users_digest", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.169124Z], timer: #Reference<0.2954531850.948174849.2616>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.255 [error] GenServer Oban.Queue.Backup.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Backup.Foreman, limit: 1, name: Oban.Queue.Backup.Producer, nonce: "u565l54h", paused: false, poll_interval: 1000, queue: "backup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.200923Z], timer: #Reference<0.2954531850.948174849.2633>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.264 [error] GenServer Oban.Queue.IngestionQueue.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.IngestionQueue.Foreman, limit: 50, name: Oban.Queue.IngestionQueue.Producer, nonce: "bv22bdbh", paused: false, poll_interval: 1000, queue: "ingestion_queue", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.219226Z], timer: #Reference<0.2954531850.948174849.2635>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.264 [error] GenServer Oban.Queue.MuteExpire.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 898ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.MuteExpire.Foreman, limit: 5, name: Oban.Queue.MuteExpire.Producer, nonce: "lah544ea", paused: false, poll_interval: 1000, queue: "mute_expire", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.219389Z], timer: #Reference<0.2954531850.948174849.2638>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.274 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "ej9qvo3c", paused: false, poll_interval: 1000, queue: "federator_outgoing", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.186360Z], timer: #Reference<0.2954531849.3982753795.94766>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.288 [error] GenServer Oban.Queue.WebPush.Producer terminating
Jan 25 23:34:44 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 899ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:44 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:44 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:44 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:44 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:44 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:44 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:44 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:44 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.WebPush.Foreman, limit: 50, name: Oban.Queue.WebPush.Producer, nonce: "dtbgi02c", paused: false, poll_interval: 1000, queue: "web_push", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:42.218644Z], timer: #Reference<0.2954531849.3982753795.94763>}
Jan 25 23:34:44 gleasonator mix[1850101]: 23:34:44.880 [info] POST /inbox
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.077 [info] POST /inbox
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.119 [error] Elixir.Pleroma.Web.MastodonAPI.StatusView failed to render {Pleroma.Web.MastodonAPI.StatusView, "show.json"}
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1946ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:112: Ecto.Repo.Queryable.one/3
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:225: Pleroma.Web.MastodonAPI.StatusView.render/2
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/status_view.ex:6: Pleroma.Web.MastodonAPI.StatusView.safe_render/3
Jan 25 23:34:46 gleasonator mix[1850101]: (elixir 1.10.4) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.123 request_id=Fl2adpdnL69L5GUExVSh [error] Elixir.Pleroma.Web.MastodonAPI.NotificationView failed to render {Pleroma.Web.MastodonAPI.NotificationView, "show.json"}
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1978ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.exists?/3
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/notification_view.ex:97: Pleroma.Web.MastodonAPI.NotificationView.render/2
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/mastodon_api/views/notification_view.ex:6: Pleroma.Web.MastodonAPI.NotificationView.safe_render/3
Jan 25 23:34:46 gleasonator mix[1850101]: (elixir 1.10.4) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.137 [warn] Instance.get_or_update_favicon("mstdn.social") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1888ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.137 request_id=Fl2aeLRYA6WZ8NUCRRRj [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1236ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.148 [error] GenServer Pleroma.Stats terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1922ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:46 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:69: Pleroma.Stats.calculate_stat_data/0
Jan 25 23:34:46 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:107: Pleroma.Stats.handle_continue/2
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:431: :gen_server.loop/7
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: {:continue, :calculate_stats}
Jan 25 23:34:46 gleasonator mix[1850101]: State: nil
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.149 [error] GenServer Oban.Queue.ScheduledActivities.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 844ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ScheduledActivities.Foreman, limit: 10, name: Oban.Queue.ScheduledActivities.Producer, nonce: "ctem92ui", paused: false, poll_interval: 1000, queue: "scheduled_activities", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.254550Z], timer: #Reference<0.2954531850.265814018.85893>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.149 [error] GenServer Oban.Queue.WebPush.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.WebPush.Foreman, limit: 50, name: Oban.Queue.WebPush.Producer, nonce: "jeuprkth", paused: false, poll_interval: 1000, queue: "web_push", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.304386Z], timer: #Reference<0.2954531850.265814018.85921>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.150 [error] GenServer Oban.Queue.IngestionQueue.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.IngestionQueue.Foreman, limit: 50, name: Oban.Queue.IngestionQueue.Producer, nonce: "4vei2cid", paused: false, poll_interval: 1000, queue: "ingestion_queue", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.290475Z], timer: #Reference<0.2954531850.265814018.85922>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.150 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "9q6snicq", paused: false, poll_interval: 1000, queue: "federator_outgoing", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.290561Z], timer: #Reference<0.2954531850.265814018.85923>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.151 [error] GenServer Oban.Queue.Mailer.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Mailer.Foreman, limit: 10, name: Oban.Queue.Mailer.Producer, nonce: "ghtilp1n", paused: false, poll_interval: 1000, queue: "mailer", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.290772Z], timer: #Reference<0.2954531850.265814018.85924>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.165 [error] GenServer Oban.Queue.Background.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Background.Foreman, limit: 5, name: Oban.Queue.Background.Producer, nonce: "mt1r78o3", paused: false, poll_interval: 1000, queue: "background", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.290882Z], timer: #Reference<0.2954531850.265814018.85925>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.165 [error] GenServer Oban.Queue.RemoteFetcher.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 888ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.RemoteFetcher.Foreman, limit: 2, name: Oban.Queue.RemoteFetcher.Producer, nonce: "duhs1jio", paused: false, poll_interval: 1000, queue: "remote_fetcher", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.142871Z], timer: #Reference<0.2954531849.3982753795.94997>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.165 [error] GenServer Oban.Queue.Backup.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Backup.Foreman, limit: 1, name: Oban.Queue.Backup.Producer, nonce: "54s0mo2n", paused: false, poll_interval: 1000, queue: "backup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.304783Z], timer: #Reference<0.2954531849.3982753795.95038>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.181 [error] GenServer Oban.Queue.ActivityExpiration.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ActivityExpiration.Foreman, limit: 10, name: Oban.Queue.ActivityExpiration.Producer, nonce: "mq7g34ca", paused: false, poll_interval: 1000, queue: "activity_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.290977Z], timer: #Reference<0.2954531850.265814018.85926>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.181 [error] GenServer Oban.Queue.AttachmentsCleanup.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.AttachmentsCleanup.Foreman, limit: 5, name: Oban.Queue.AttachmentsCleanup.Producer, nonce: "c7hd82kq", paused: false, poll_interval: 1000, queue: "attachments_cleanup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291079Z], timer: #Reference<0.2954531850.265814018.85927>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.182 [error] GenServer Oban.Queue.TokenExpiration.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.TokenExpiration.Foreman, limit: 5, name: Oban.Queue.TokenExpiration.Producer, nonce: "dalkb6sj", paused: false, poll_interval: 1000, queue: "token_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291186Z], timer: #Reference<0.2954531850.265814018.85928>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.182 [error] GenServer Oban.Queue.FederatorIncoming.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorIncoming.Foreman, limit: 50, name: Oban.Queue.FederatorIncoming.Producer, nonce: "7e714ko3", paused: false, poll_interval: 1000, queue: "federator_incoming", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291287Z], timer: #Reference<0.2954531850.265814018.85929>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.183 [error] GenServer Oban.Queue.Transmogrifier.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Transmogrifier.Foreman, limit: 20, name: Oban.Queue.Transmogrifier.Producer, nonce: "hl19f1pq", paused: false, poll_interval: 1000, queue: "transmogrifier", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291380Z], timer: #Reference<0.2954531850.265814018.85930>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.195 [error] GenServer Oban.Queue.MuteExpire.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.MuteExpire.Foreman, limit: 5, name: Oban.Queue.MuteExpire.Producer, nonce: "9m3sopma", paused: false, poll_interval: 1000, queue: "mute_expire", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291731Z], timer: #Reference<0.2954531850.265814018.85932>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.205 [error] #PID<0.13253.1165> running Pleroma.Web.Endpoint (connection #PID<0.29685.1164>, stream id 1) terminated
Jan 25 23:34:46 gleasonator mix[1850101]: Server: gleasonator.com:80 (http)
Jan 25 23:34:46 gleasonator mix[1850101]: Request: POST /inbox
Jan 25 23:34:46 gleasonator mix[1850101]: ** (exit) an exception was raised:
Jan 25 23:34:46 gleasonator mix[1850101]: ** (FunctionClauseError) no function clause matching in Plug.Conn.resp/3
Jan 25 23:34:46 gleasonator mix[1850101]: (plug 1.10.4) lib/plug/conn.ex:580: Plug.Conn.resp(%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{digest: "SHA-256=VxxPH04860XbIBVGoi6gNnrJMNNSmA4eCKwokgtCS6M=", kind: :error, layout: false, locale: "en", reason: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1236ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}, remote_ip_found: true, stack: [{Ecto.Adapters.SQL, :raise_sql_call_error, 1, [file: 'lib/ecto/adapters/sql.ex', line: 593]}, {Ecto.Adapters.SQL, :execute, 5, [file: 'lib/ecto/adapters/sql.ex', line: 526]}, {Ecto.Repo.Queryable, :execute, 4, [file: 'lib/ecto/repo/queryable.ex', line: 192]}, {Ecto.Repo.Queryable, :all, 3, [file: 'lib/ecto/repo/queryable.ex', line: 17]}, {Ecto.Repo.Queryable, :one, 3, [file: 'lib/ecto/repo/queryable.ex', line: 112]}, {Pleroma.Instances.Instance, :set_reachable, 1, [file: 'lib/pleroma/instances/instance.ex', line: 84]}, {Pleroma.Web.ActivityPub.ActivityPubController, :set_requester_reachable, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 515]}, {Pleroma.Web.ActivityPub.ActivityPubController, :phoenix_controller_pipeline, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 5]}], status: 500, token: nil, user: %Pleroma.User{password_confirmation: nil, keys: "-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEAsE5veCPsyjFfzbinpevw+XWfkIAw/z127WKlTZ+lRndTtrgy\nG0S3zxU1o+iRSPchpcE2J+0HrKEoCrsUm1YSRVAXXOZLMNOX6X5B2dNQHch3oFcv\nuJTz1DdyQOkiCdqs9sKywlZDqzr8z82tjWkjJoSYUcp5kYxL9yoIgDerSWRzoxTH\nzU1pEvFm9ihSP/QKBoXCd+XqPW8tqp5t6xotQ+sPrOLtwoU/0UGto3wsqZj7FXoh\nDOvlGK10qZ2bPUFQN1hyABo/UviohoYtxvbEVJRpqlx662rkrsMHnJrSDyNartDk\nRfd7bv/+KgL0jcrVbao/GOdSu+sNDcsAyPojTQIDAQABAoIBAQCrGdPQMn0w1K33\nRodc3Rel4WMTp3TloE70nlFyfca2FkQmBaiLEiuYSysdgSzONRhwdowKP1y9SkFD\n+L9M6UEeB1ZFEDJUlmTNc3yQ2yTbYpvF/hcP+W6Qu6akHByul4xYhidZtsJBQSPs\nv9z4adp1HSV96xQEDrYCjsZlOKGAvODuLXtBsKGFR6+0ofOFwDPHBkx/pi+5zAYZ\ngiSQUOipOU78t4cP1gy+f5q+C+XbmQZl7SepdZiTxmBoS5S7C4qm8EaGxWxdW6/j\nwZCATVY3/j7zsI7cf8D591/q8N0DCrnEsJUgx6UqNuBj4hV3nkzjur54Qn+3D+jZ\nxlodeRHBAoGBAOF22lHz+JMIYIGqORMFySQ5VsPFLwfMPH8L+9kqnL9v3w/YEl8/\n5XbsjyNrqkbDwEq9/bdR7z/CT3Bt0/hRs6z2pAVT7kgHsle3ymOLBEOThIl3Ftgq\n718z0hQ1ZtBQjt+WMI43R5CIa44uqehiTlEfw4jCis9Eh4nMnnAH4TAZAoGBAMgv\nNpk9h+zKhdz2qWE/e1HWIHuJ5dfS/8o/1ipidZWWR2RP49ZkTZt9IclplTNyNCll\nDVkUlkoHLtm5/Q5ajfHgMmj9TOsaZP1KWC1vhEFYtQsK/+E3UrWvZN01NYlZHjrw\nhw05FW/nhYKbSuf8bTpKuyDgOQuJzwu4cm6v5+NVAoGBALQiqnF51CtdMQXIA8uv\n/rGDYlANlfLL+bWj1HcAtbW54uUcxCj3uxTWEdLB7YyWT2cn/rLdsqIlVCTGiunG\nFSDm+csL8L/57kJpJDl9001OakEONeFGBHPyyjC9LcTX6B65N1gY0IOoiw5ROAzj\npiDomOgGTrMk1ImDQz7M/NzZAoGAPAcMVD9DNxozseK1P9gNXqja7LCZH3LCCR/D\nVMpZgBWkKsxaAbvqGaoYK5FhkB+6zH7k1cV6m5Az2E4RN/Cnh9Q2yYzLN+SeS+Wz\nM0yxV/cFst4FhL9WKe0CTeLavujeS8Y8eRaY+wlt25A9kfr7tfMa0ezqri8Qyu7w\nQYi1/FECgYEAiU+UXxASRQtlmhLf+bzZ4HXYBxkn5KzKwZffn96IAbU+sKxguTcu\n1OwCIGCLirrGl3ZC7rWwb2VKwU+2kFqfQAqMan+aGmXEo8NSQkMx/gSnWThns7M0\nsTCaQaWdh2TG6fxGdg+0m4JN6m5lDCiKlRhfQ5XGOUTiI/rgC1D1uXk=\n-----END RSA PRIVATE KEY-----", muted_users: #Ecto.Association.NotLoaded<association :muted_users is not loaded>, name: "Stretch", uri: "https://bikeshed.party/users/feld", mutes: [], confirmation_token: nil, hide_follows: true, fields: [%{"name" => "Threema", "value" => "<a href=\"https://threema.id/SNEBN32Y\">https://threema.id/SNEBN32Y</a>"}, %{"name" => "Telegram", "value" => "<a href=\"https://t.me/feldpos\">https://t.me/feldpos</a>"}, %{"name" => "My Links", "value" => "<a href=\"https://allmylinks.com/feld\">https://allmylinks.com/feld</a>"}, %{"name" => "Librio.fm Referral", "value" => "<a href=\"https://libro.fm/referral?rf_code=lfm151943\">https://libro.fm/referral?rf_code=lfm151943</a>"}], password_reset_pending: false, subscribee_users: #Ecto.Association.NotLoaded<association :subscribee_users is not loaded>, muted_notifications: [], reblog_mutee_mutes: #Ecto.Association.NotLoaded<association :reblog_mutee_mutes is not loaded>, avatar: %{"type" => "Image", "url" => [%{"href" => "https://media.bikeshed.party/pleroma/7785d1a2062924ef90ec9b9366b3fcb4d8e35791227ab01e88a827fbba969c41.blob"}]}, note_count: 11667, updated_at: ~N[2021-01-25 15:16:44], registrations: #Ecto.Association.NotLoaded<association :registrations is not loaded>, nickname: "feld@bikeshed.party", background: %{}, following_address: "https://bikeshed.party/users/feld/following", confirmation_pending: true, follower_count: 1358, multi_factor_authentication_settings: %Pleroma.MFA.Settings{backup_codes: [], enabled: false, totp: nil}, muter_users: #Ecto.Association.NotLoaded<association :muter_users is not loaded>, pinned_activities: [], following_count: 507, search_type: nil, hide_followers: false, default_scope: "public", invisible: false, follower_address: "https://bikeshed.party/users/feld/followers", is_locked: false, inserted_at: ~N[2020-01-09 02:56:17], shared_inbox: "https://bikeshed.party/inbox", email_notifications: %{"digest" => false}, muted_reblogs: [], tags: [], reblog_muter_mutes: #Ecto.Association.NotLoaded<association :reblog_muter_mutes is not loaded>, ...}, valid_signature: true}, before_send: [#Function<0.119468924/1 in Pleroma.Web.Endpoint.PipelineInstrumenter.call/2>, #Function<1.132129851/1 in Plug.Logger.call/2>, #Function<0.60895335/1 in Plug.Telemetry.call/2>], body_params: %{"@context" => ["https://www.w3.org/ns/activitystreams", "https://bikeshed.party/schemas/litepub-0.1.jsonld", %{"@language" => "und"}], "actor" => "https://bikeshed.party/users/feld", "cc" => ["https://bikeshed.party/users/feld/followers"], "context" => "tag:mastodon.social,2021-01-25:objectId=218936100:objectType=Conversation", "context_id" => 21758175, "directMessage" => false, "expires_at" => "2021-03-26T22:34:43.783766Z", "id" => "https://bikeshed.party/activities/e34d2b29-6a93-4b1a-b54e-9dcf529cf615", "object" => %{"actor" => "https://bikeshed.party/users/feld", "attachment" => [%{"mediaType" => "image/png", "name" => "", "type" => "Document", "url" => "https://media.bikeshed.party/pleroma/43fcca46b13e2948975b23c30b2310d874992600c3ba192492a455c6c232af6b.png"}], "attributedTo" => "https://bikeshed.party/users/feld", "cc" => ["https://bikeshed.party/users/feld/followers"], "content" => "<span class=\"h-card\"><a class=\"u-url mention\" data-user=\"24017\" href=\"https://mastodon.social/@vogon\" rel=\"ugc\">@<span>vogon</span></a></span> not entirely true, they started building their own ventilators using the Model3/Y MCU. But that project seemingly fizzled out. Here's a photo.", "context" => "tag:mastodon.social,2021-01-25:objectId=218936100:objectType=Conversation", "conversation" => "tag:mastodon.social,2021-01-25:objectId=218936100:objectType=Conversation", "id" => "https://bikeshed.party/objects/b3364ec7-90f6-46a7-b183-b10d4497acf2", "inReplyTo" => "https://mastodon.social/users/vogon/statuses/105618719152936072", "published" => "2021-01-25T22:34:43.783616Z", "sensitive" => false, "source" => "@vogon@mastodon.social not entirely true, they started building their own ventilators using the Model3/Y MCU. But that project seemingly fizzled out. Here's a photo.", "summary" => "re: coronavirus", "tag" => [%{"href" => "https://mastodon.social/users/vogon", "name" => "@vogon@mastodon.social", "type" => "Mention"}], "to" => ["https://mastodon.social/users/vogon", "https://www.w3.org/ns/activitystreams#Public"], "type" => "Note"}, "published" => "2021-01-25T22:34:43.7830 (truncated)
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.206 [error] GenServer Oban.Queue.NewUsersDigest.Producer terminating
Jan 25 23:34:46 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:46 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:46 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:46 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:46 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:46 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:46 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:46 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:46 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.NewUsersDigest.Foreman, limit: 1, name: Oban.Queue.NewUsersDigest.Producer, nonce: "hqi3mtpk", paused: false, poll_interval: 1000, queue: "new_users_digest", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:44.291573Z], timer: #Reference<0.2954531850.265814018.85931>}
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.637 [info] GET /home
Jan 25 23:34:46 gleasonator mix[1850101]: 23:34:46.844 request_id=Fl2adalMEhryNtIDUJWC [info] Sent 200 in 15033ms
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.501 request_id=Fl2addIRxh8H9NYExVBh [info] Sent 200 in 15008ms
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.502 [info] Postgrex.Protocol (#PID<0.19963.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.7955.1165> exited
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.512 [error] GenServer Oban.Queue.NewUsersDigest.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 303ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.NewUsersDigest.Foreman, limit: 1, name: Oban.Queue.NewUsersDigest.Producer, nonce: "j7l18qbf", paused: false, poll_interval: 1000, queue: "new_users_digest", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.207040Z], timer: #Reference<0.2954531849.3982753795.95419>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.512 request_id=Fl2aeR0Pb4yUuRgCRRVj [error] Internal server error: %Cachex.ExecutionError{message: "connection not available and request was dropped from queue after 872ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n"}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.512 request_id=Fl2aeR0Pb4yUuRgCRRVj [info] Sent 500 in 875ms
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.513 [error] #PID<0.26088.1164> running Pleroma.Web.Endpoint (connection #PID<0.8950.1165>, stream id 1) terminated
Jan 25 23:34:47 gleasonator mix[1850101]: Server: gleasonator.com:80 (http)
Jan 25 23:34:47 gleasonator mix[1850101]: Request: GET /home
Jan 25 23:34:47 gleasonator mix[1850101]: ** (exit) an exception was raised:
Jan 25 23:34:47 gleasonator mix[1850101]: ** (Cachex.ExecutionError) connection not available and request was dropped from queue after 872ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (cachex 3.2.0) lib/cachex.ex:1438: Cachex.unwrap_unsafe/1
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/user.ex:1107: Pleroma.User.get_cached_by_id/1
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/user.ex:1137: Pleroma.User.get_cached_by_nickname_or_id/2
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/fallback/redirect_controller.ex:27: Pleroma.Web.Fallback.RedirectController.redirector_with_meta/2
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/fallback/redirect_controller.ex:5: Pleroma.Web.Fallback.RedirectController.action/2
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/fallback/redirect_controller.ex:5: Pleroma.Web.Fallback.RedirectController.phoenix_controller_pipeline/2
Jan 25 23:34:47 gleasonator mix[1850101]: (phoenix 1.5.6) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/web/endpoint.ex:5: Pleroma.Web.Endpoint.plug_builder_call/2
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.527 request_id=Fl2adpdnL69L5GUExVSh [warn] Instance.get_or_update_favicon("freespeechextremist.com") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1386ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.536 [warn] Instance.get_or_update_favicon("poa.st") error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1278ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.537 [error] GenServer Oban.Queue.WebPush.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 344ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.WebPush.Foreman, limit: 50, name: Oban.Queue.WebPush.Producer, nonce: "rhu5551a", paused: false, poll_interval: 1000, queue: "web_push", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.165502Z], timer: #Reference<0.2954531850.265814018.86088>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.537 [error] GenServer Oban.Queue.RemoteFetcher.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 304ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.RemoteFetcher.Foreman, limit: 2, name: Oban.Queue.RemoteFetcher.Producer, nonce: "5o2r7idt", paused: false, poll_interval: 1000, queue: "remote_fetcher", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.180208Z], timer: #Reference<0.2954531850.265814018.86105>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.537 [error] GenServer Oban.Queue.Backup.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 304ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Backup.Foreman, limit: 1, name: Oban.Queue.Backup.Producer, nonce: "76sa8ib2", paused: false, poll_interval: 1000, queue: "backup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.180255Z], timer: #Reference<0.2954531850.265814018.86106>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.539 request_id=Fl2aePuwCPWRM9cCRRTj [info] Sent 200 in 1462ms
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.540 [error] GenServer Pleroma.Stats terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1346ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Jan 25 23:34:47 gleasonator mix[1850101]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Jan 25 23:34:47 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Jan 25 23:34:47 gleasonator mix[1850101]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:69: Pleroma.Stats.calculate_stat_data/0
Jan 25 23:34:47 gleasonator mix[1850101]: (pleroma 2.2.50-586-gafc68a05-develop) lib/pleroma/stats.ex:107: Pleroma.Stats.handle_continue/2
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:431: :gen_server.loop/7
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: {:continue, :calculate_stats}
Jan 25 23:34:47 gleasonator mix[1850101]: State: nil
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.550 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 315ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "a5nlm4qr", paused: false, poll_interval: 1000, queue: "federator_outgoing", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.151054Z], timer: #Reference<0.2954531850.948174849.3239>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.550 [error] GenServer Oban.Queue.Mailer.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 315ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Mailer.Foreman, limit: 10, name: Oban.Queue.Mailer.Producer, nonce: "fiotg4km", paused: false, poll_interval: 1000, queue: "mailer", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.151729Z], timer: #Reference<0.2954531850.948174849.3240>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.551 [error] GenServer Oban.Queue.ActivityExpiration.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ActivityExpiration.Foreman, limit: 10, name: Oban.Queue.ActivityExpiration.Producer, nonce: "gtqhskt5", paused: false, poll_interval: 1000, queue: "activity_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195212Z], timer: #Reference<0.2954531850.948174849.3267>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.565 [error] GenServer Oban.Queue.IngestionQueue.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 315ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.IngestionQueue.Foreman, limit: 50, name: Oban.Queue.IngestionQueue.Producer, nonce: "4jpgjo2j", paused: false, poll_interval: 1000, queue: "ingestion_queue", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.150408Z], timer: #Reference<0.2954531850.948174849.3238>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.565 [error] GenServer Oban.Queue.Background.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Background.Foreman, limit: 5, name: Oban.Queue.Background.Producer, nonce: "dhddkcdj", paused: false, poll_interval: 1000, queue: "background", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195176Z], timer: #Reference<0.2954531850.948174849.3266>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.566 [error] GenServer Oban.Queue.TokenExpiration.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.TokenExpiration.Foreman, limit: 5, name: Oban.Queue.TokenExpiration.Producer, nonce: "3vkhddcv", paused: false, poll_interval: 1000, queue: "token_expiration", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195249Z], timer: #Reference<0.2954531850.948174849.3269>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.567 [error] GenServer Oban.Queue.FederatorIncoming.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.FederatorIncoming.Foreman, limit: 50, name: Oban.Queue.FederatorIncoming.Producer, nonce: "u60hqoq4", paused: false, poll_interval: 1000, queue: "federator_incoming", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195265Z], timer: #Reference<0.2954531850.948174849.3270>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.571 [error] GenServer Oban.Queue.MuteExpire.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.MuteExpire.Foreman, limit: 5, name: Oban.Queue.MuteExpire.Producer, nonce: "2ubl94g5", paused: false, poll_interval: 1000, queue: "mute_expire", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195532Z], timer: #Reference<0.2954531850.948174849.3272>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.571 [error] GenServer Oban.Queue.ScheduledActivities.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.ScheduledActivities.Foreman, limit: 10, name: Oban.Queue.ScheduledActivities.Producer, nonce: "ihmri1vc", paused: false, poll_interval: 1000, queue: "scheduled_activities", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195626Z], timer: #Reference<0.2954531850.948174849.3273>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.584 [error] GenServer Oban.Queue.AttachmentsCleanup.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.AttachmentsCleanup.Foreman, limit: 5, name: Oban.Queue.AttachmentsCleanup.Producer, nonce: "92s482sl", paused: false, poll_interval: 1000, queue: "attachments_cleanup", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195231Z], timer: #Reference<0.2954531850.948174849.3268>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.591 [error] GenServer Oban.Queue.Transmogrifier.Producer terminating
Jan 25 23:34:47 gleasonator mix[1850101]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 314ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Jan 25 23:34:47 gleasonator mix[1850101]: 1. By tracking down slow queries and making sure they are running fast enough
Jan 25 23:34:47 gleasonator mix[1850101]: 2. Increasing the pool_size (albeit it increases resource consumption)
Jan 25 23:34:47 gleasonator mix[1850101]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Jan 25 23:34:47 gleasonator mix[1850101]: See DBConnection.start_link/2 for more information
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Jan 25 23:34:47 gleasonator mix[1850101]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Jan 25 23:34:47 gleasonator mix[1850101]: Last message: :poll
Jan 25 23:34:47 gleasonator mix[1850101]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "gleasonator", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], backup: [limit: 1], federator_incoming: [limit: 50], federator_outgoing: [limit: 50], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 5], new_users_digest: [limit: 1], mute_expire: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: Oban.Queue.Transmogrifier.Foreman, limit: 20, name: Oban.Queue.Transmogrifier.Producer, nonce: "1k4bistc", paused: false, poll_interval: 1000, queue: "transmogrifier", reset_timer: nil, running: %{}, started_at: ~U[2021-01-25 22:34:46.195282Z], timer: #Reference<0.2954531850.948174849.3271>}
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.695 request_id=Fl2addvebNYQPnsCRQlD [info] Sent 200 in 15036ms
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.696 [error] Postgrex.Protocol (#PID<0.3197.265>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.5071.1165> timed out because it queued and checked out the connection for longer than 15000ms
Jan 25 23:34:47 gleasonator mix[1850101]: #PID<0.5071.1165> was at location:
Jan 25 23:34:47 gleasonator mix[1850101]: :prim_inet.recv0/3
Jan 25 23:34:47 gleasonator mix[1850101]: (postgrex 0.15.6) lib/postgrex/protocol.ex:2838: Postgrex.Protocol.msg_recv/4
Jan 25 23:34:47 gleasonator mix[1850101]: (postgrex 0.15.6) lib/postgrex/protocol.ex:1880: Postgrex.Protocol.recv_bind/3
Jan 25 23:34:47 gleasonator mix[1850101]: (postgrex 0.15.6) lib/postgrex/protocol.ex:1735: Postgrex.Protocol.bind_execute_close/4
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection/holder.ex:316: DBConnection.Holder.holder_apply/4
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:1255: DBConnection.run_execute/5
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:1342: DBConnection.run/6
Jan 25 23:34:47 gleasonator mix[1850101]: (db_connection 2.2.2) lib/db_connection.ex:595: DBConnection.execute/4
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.816 [info] Postgrex.Protocol (#PID<0.29141.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.14977.1165> exited
Jan 25 23:34:47 gleasonator mix[1850101]: 23:34:47.818 request_id=Fl2adeO5KZlw4X0DUJXC [info] Sent 200 in 15027ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.108 request_id=Fl2adsDZgqRoQVQCRQ1j [info] Sent 200 in 11607ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.572 [error] Postgrex.Protocol (#PID<0.2840.265>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.23544.1165> timed out because it queued and checked out the connection for longer than 15000ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.574 request_id=Fl2adhBnz2vZlv8ExVFB [info] Sent 200 in 15034ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.644 [info] Postgrex.Protocol (#PID<0.32394.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.16045.1164> exited
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.645 request_id=Fl2adhZcHdytjJIExVGB [info] Sent 200 in 15005ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.754 request_id=Fl2adpdnL69L5GUExVSh [info] Sent 200 in 12949ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.779 request_id=Fl2adht1nMJAA90DUJeC [info] Sent 200 in 15054ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.780 [info] Postgrex.Protocol (#PID<0.30466.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.32446.1164> exited
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.877 request_id=Fl2adiOhoIMlj3MExVIB [info] Sent 200 in 15014ms
Jan 25 23:34:48 gleasonator mix[1850101]: 23:34:48.878 [info] Postgrex.Protocol (#PID<0.30808.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.7295.1165> exited
Jan 25 23:34:52 gleasonator mix[1850101]: 23:34:52.573 [info] Postgrex.Protocol (#PID<0.30908.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.19556.1165> exited
Jan 25 23:34:52 gleasonator mix[1850101]: 23:34:52.574 [info] Postgrex.Protocol (#PID<0.2973.265>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.2923.1165> exited
Jan 25 23:34:52 gleasonator mix[1850101]: 23:34:52.577 [info] Postgrex.Protocol (#PID<0.31792.264>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.21935.1165> exited
Jan 25 23:34:52 gleasonator mix[1850101]: 23:34:52.683 [info] Application pleroma exited: shutdown
To recover I ran systemctl restart pleroma
Not totally sure how to decipher this yet. It's running out of database connections, but why? I'm running the develop-ish branch.
Edited by Alex Gleason