Things start timing out often on 2.1
I've twice now woken up to my instance not working, log is full of oban errors, and the memory use seems to be full so it's probably that. However the machine has 2gb of ram and I haven't seen it run out or the same issues before.
Some of the logs
Aug 30 07:47:21 shpposterclub.localdomain systemd-journald[739]: Suppressed 1574 messages from /system.slice/pleroma.service
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.676 request_id=Fi_1mznV5LSmmSAAJmsR [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1873ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.676 request_id=Fi_1mznddoVbLXcAJmsh [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 1873ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.679 request_id=Fi_1m3SeBrPXkN4AJmwR [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 891ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.679 request_id=Fi_1m3V_WOnwkhoAJmwx [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 876ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.679 request_id=Fi_1m5qnxtSEIp8AJmxR [error] Internal server error: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 254ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. By tracking down slow queries and making sure they are running fast enough\n 2. Increasing the pool_size (albeit it increases resource consumption)\n 3. Allow requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.680 request_id=Fi_1mznV5LSmmSAAJmsR [info] Converted error DBConnection.ConnectionError to 500 response
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.681 request_id=Fi_1mznddoVbLXcAJmsh [info] Converted error DBConnection.ConnectionError to 500 response
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.684 request_id=Fi_1m3SeBrPXkN4AJmwR [info] Converted error DBConnection.ConnectionError to 500 response
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.684 request_id=Fi_1m3V_WOnwkhoAJmwx [info] Converted error DBConnection.ConnectionError to 500 response
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.684 request_id=Fi_1m5qnxtSEIp8AJmxR [info] Converted error DBConnection.ConnectionError to 500 response
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.687 [error] GenServer Oban.Queue.AttachmentsCleanup.Producer terminating
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 961ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 1. By tracking down slow queries and making sure they are running fast enough
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 2. Increasing the pool_size (albeit it increases resource consumption)
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: See DBConnection.start_link/2 for more information
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: Last message: :poll
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.ClearOauthTokenWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [:*], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.StatsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [:*], minutes: [:*], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "shpposterclub", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5, remote_fetcher: 2, attachments_cleanup: 5, new_users_digest: 1], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatched_at: nil, foreman: Oban.Queue.AttachmentsCleanup.Foreman, limit: 5, name: Oban.Queue.AttachmentsCleanup.Producer, nonce: "f2l8tffd", paused: false, queue: "attachments_cleanup", reset_timer: nil, running: %{}, started_at: ~U[2020-08-30 05:47:19.683741Z], timer: #Reference<0.1319412876.1212416001.53537>}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.688 [error] GenServer Oban.Queue.FederatorOutgoing.Producer terminating
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 961ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 1. By tracking down slow queries and making sure they are running fast enough
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 2. Increasing the pool_size (albeit it increases resource consumption)
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 3. Allow requests to wait longer by increasing :queue_target and :queue_interval
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: See DBConnection.start_link/2 for more information
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (db_connection 2.2.2) lib/db_connection.ex:745: DBConnection.run/3
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: Last message: :poll
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.ClearOauthTokenWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [:*], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.StatsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [:*], minutes: [:*], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, log: false, name: Oban, node: "shpposterclub", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: 10, federator_incoming: 50, federator_outgoing: 50, web_push: 50, mailer: 10, transmogrifier: 20, scheduled_activities: 10, background: 5, remote_fetcher: 2, attachments_cleanup: 5, new_users_digest: 1], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatched_at: nil, foreman: Oban.Queue.FederatorOutgoing.Foreman, limit: 50, name: Oban.Queue.FederatorOutgoing.Producer, nonce: "5mfnesqh", paused: false, queue: "federator_outgoing", reset_timer: nil, running: %{}, started_at: ~U[2020-08-30 05:47:19.691907Z], timer: #Reference<0.1319412876.1212416001.53538>}
Aug 30 07:47:21 shpposterclub.localdomain mix[32475]: 07:47:21.689 [error] GenServer Oban.Queue.Mailer.Producer terminating
Edited by Shpuld Shpludson