Crashes after 24 hours
Precheck
I've commented on a few issues I thought was relevant but it either wasn't applicable or the issue was closed without resolution.
Environment
- Docker image
- Backend version: 2.4.52-665-ga022b9d7
- Frontend version: b13d8f7e
- Elixir image
- PostgreSQL 14
Bug description
I've had to restart the server every 24 hours to resume operation. Before restarting I've noted that resource usage is normal, the server is not out of memory or storage. I was having this issue before I updated postgres to 14 from 12. I restarted with a fresh database when I updated postgres.
pleroma | 06:39:11.405 [error] GenServer {Oban.Registry, {Oban, {:producer, "remote_fetcher"}}} terminating
pleroma | ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1406ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
pleroma |
pleroma | 1. Ensuring your database is available and that you can connect to it
pleroma | 2. Tracking down slow queries and making sure they are running fast enough
pleroma | 3. Increasing the pool_size (albeit it increases resource consumption)
pleroma | 4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
pleroma |
pleroma | See DBConnection.start_link/2 for more information
pleroma |
pleroma | (db_connection) lib/db_connection.ex:784: DBConnection.run/3
pleroma | (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
pleroma | (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
pleroma | (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
pleroma | Last message: :poll
pleroma | State: %Oban.Queue.Producer.State{circuit: :enabled, conf: %Oban.Config{circuit_backoff: 30000, crontab: [{%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [0]}, Pleroma.Workers.Cron.DigestEmailsWorker, []}, {%Oban.Crontab.Cron{days: [:*], hours: [0], minutes: [0], months: [:*], reboot: false, weekdays: [:*]}, Pleroma.Workers.Cron.NewUsersDigestWorker, []}], dispatch_cooldown: 5, get_dynamic_repo: nil, log: false, name: Oban, node: "pleroma@127.0.0.1", plugins: [Oban.Plugins.Pruner], poll_interval: 1000, prefix: "public", queues: [activity_expiration: [limit: 10], token_expiration: [limit: 5], filter_expiration: [limit: 1], backup: [limit: 1], ingestion_queue: [limit: 50], web_push: [limit: 50], mailer: [limit: 10], transmogrifier: [limit: 20], scheduled_activities: [limit: 10], poll_notifications: [limit: 10], background: [limit: 5], remote_fetcher: [limit: 2], attachments_cleanup: [limit: 1], new_users_digest: [limit: 1], mute_expire: [limit: 5], federator_incoming: [limit: 5], federator_outgoing: [limit: 5]], repo: Pleroma.Repo, shutdown_grace_period: 15000, timezone: "Etc/UTC"}, cooldown_ref: nil, dispatch_cooldown: 5, dispatched_at: nil, foreman: {:via, Registry, {Oban.Registry, {Oban, {:foreman, "remote_fetcher"}}}}, limit: 2, name: {:via, Registry, {Oban.Registry, {Oban, {:producer, "remote_fetcher"}}}}, nonce: "a4434vm4", paused: false, poll_interval: 1000, queue: "remote_fetcher", reset_timer: nil, running: %{}, started_at: ~U[2022-09-21 06:39:08.797511Z], timer: #Reference<0.582941313.985137153.190663>}
pleroma |
pleroma | 06:39:11.727 [info] Application pleroma exited: shutdown
postgres_1 | 2022-09-21 06:38:53.891 UTC [3511] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:38:54.503 UTC [3180] ERROR: canceling statement due to user request
postgres_1 | 2022-09-21 06:38:54.503 UTC [3180] STATEMENT: UPDATE "public"."oban_jobs" AS o0 SET "state" = $1 WHERE (o0."id" IN (SELECT so0."id" FROM "public"."oban_jobs" AS so0 WHERE (so0."state" IN ('scheduled','retryable')) AND (so0."queue" = $2) AND (so0."scheduled_at" <= $3) FOR UPDATE SKIP LOCKED))
postgres_1 | 2022-09-21 06:38:54.090 UTC [3511] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:38:54.554 UTC [56] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:38:54.550 UTC [2732] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:38:56.084 UTC [56] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:38:57.359 UTC [2732] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:38:57.323 UTC [2734] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:39:01.301 UTC [2733] ERROR: canceling statement due to user request
postgres_1 | 2022-09-21 06:39:01.301 UTC [2733] STATEMENT: UPDATE "public"."oban_jobs" AS o0 SET "state" = $1 WHERE (o0."id" IN (SELECT so0."id" FROM "public"."oban_jobs" AS so0 WHERE (so0."state" IN ('scheduled','retryable')) AND (so0."queue" = $2) AND (so0."scheduled_at" <= $3) FOR UPDATE SKIP LOCKED))
postgres_1 | 2022-09-21 06:38:58.512 UTC [2734] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:39:04.599 UTC [3180] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:39:07.557 UTC [3180] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:39:07.630 UTC [2733] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:39:08.148 UTC [2733] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:39:08.200 UTC [3512] ERROR: canceling statement due to user request
postgres_1 | 2022-09-21 06:39:08.200 UTC [3512] STATEMENT: UPDATE "public"."oban_jobs" AS o0 SET "state" = $1 WHERE (o0."id" IN (SELECT so0."id" FROM "public"."oban_jobs" AS so0 WHERE (so0."state" IN ('scheduled','retryable')) AND (so0."queue" = $2) AND (so0."scheduled_at" <= $3) FOR UPDATE SKIP LOCKED))
postgres_1 | 2022-09-21 06:39:09.050 UTC [3181] ERROR: canceling statement due to user request
postgres_1 | 2022-09-21 06:39:09.050 UTC [3181] STATEMENT: DELETE FROM "public"."oban_jobs" AS o0 USING (SELECT so0."id" AS "id" FROM "public"."oban_jobs" AS so0 WHERE (so0."state" IN ('completed','cancelled','discarded')) AND (so0."attempted_at" < $1) LIMIT $2) AS s1 WHERE (o0."id" = s1."id")
postgres_1 | 2022-09-21 06:39:09.837 UTC [3512] LOG: could not send data to client: Broken pipe
postgres_1 | 2022-09-21 06:39:09.865 UTC [3512] FATAL: connection to client lost
postgres_1 | 2022-09-21 06:39:10.123 UTC [3179] ERROR: canceling statement due to user request
postgres_1 | 2022-09-21 06:39:10.123 UTC [3179] STATEMENT: UPDATE "public"."oban_jobs" AS o0 SET "state" = $1, "attempted_at" = $2, "attempted_by" = $3, "attempt" = o0."attempt" + $4 WHERE (o0."id" IN (SELECT so0."id" FROM "public"."oban_jobs" AS so0 WHERE (so0."state" = 'available') AND (so0."queue" = $5) ORDER BY so0."priority", so0."scheduled_at", so0."id" LIMIT $6 FOR UPDATE SKIP LOCKED)) RETURNING o0."id", o0."state", o0."queue", o0."worker", o0."args", o0."errors", o0."tags", o0."attempt", o0."attempted_by", o0."max_attempts", o0."meta", o0."priority", o0."attempted_at", o0."completed_at", o0."discarded_at", o0."cancelled_at", o0."inserted_at", o0."scheduled_at"
postgres_1 | 2022-09-21 06:39:11.189 UTC [3568] LOG: PID 3558 in cancel request did not match any process