Can't load notifications endpoint after disk space error while running 2.3.0 migrations
This is likely a PEBKAC issue, but I'm not sure how to unbreak my instance now.
I just upgraded from 2.2.2 to 2.3.0, and while running migrations my filesystem filled up. I cleared some space and re-ran the migrations, but it seems something is still broken. I tried rolling back the upgrade, but nothing I tried worked.
I'm running an OTP install on Ubuntu 20.04, and my logs are filling up with errors like the following:
Mar 08 16:14:39 procyon pleroma[1777730]: 16:14:39.349 [error] Postgrex.Protocol (#PID<0.6574.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.8887.0> timed out because it queued and checked out the connection for longer than 15000ms
Mar 08 16:14:39 procyon pleroma[1777730]: #PID<0.8887.0> was at location:
Mar 08 16:14:39 procyon pleroma[1777730]: :prim_inet.recv0/3
Mar 08 16:14:39 procyon pleroma[1777730]: (postgrex 0.15.7) lib/postgrex/protocol.ex:2838: Postgrex.Protocol.msg_recv/4
Mar 08 16:14:39 procyon pleroma[1777730]: (postgrex 0.15.7) lib/postgrex/protocol.ex:1880: Postgrex.Protocol.recv_bind/3
Mar 08 16:14:39 procyon pleroma[1777730]: (postgrex 0.15.7) lib/postgrex/protocol.ex:1735: Postgrex.Protocol.bind_execute_close/4
Mar 08 16:14:39 procyon pleroma[1777730]: (db_connection 2.3.1) lib/db_connection/holder.ex:316: DBConnection.Holder.holder_apply/4
Mar 08 16:14:39 procyon pleroma[1777730]: (db_connection 2.3.1) lib/db_connection.ex:1272: DBConnection.run_execute/5
Mar 08 16:14:39 procyon pleroma[1777730]: (db_connection 2.3.1) lib/db_connection.ex:1359: DBConnection.run/6
Mar 08 16:14:39 procyon pleroma[1777730]: (db_connection 2.3.1) lib/db_connection.ex:557: DBConnection.parsed_prepare_execute/5
Mar 08 16:14:39 procyon pleroma[1777730]: 16:14:39.350 request_id=FmpqFzlJUvVGM1gAABFS [error] Internal server error: %DBConnection.ConnectionError{message: "tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)", reason: :error, severity: :error}
Mar 08 16:14:39 procyon pleroma[1777730]: 16:14:39.362 [error] #PID<0.8887.0> running Pleroma.Web.Endpoint (connection #PID<0.8886.0>, stream id 1) terminated
Mar 08 16:14:39 procyon pleroma[1777730]: Server: squ.alid.pw:80 (http)
Mar 08 16:14:39 procyon pleroma[1777730]: Request: GET /api/v1/notifications?with_muted=true&limit=20
Mar 08 16:14:39 procyon pleroma[1777730]: ** (exit) an exception was raised:
Mar 08 16:14:39 procyon pleroma[1777730]: ** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
Mar 08 16:14:39 procyon pleroma[1777730]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:593: Ecto.Adapters.SQL.raise_sql_call_error/1
Mar 08 16:14:39 procyon pleroma[1777730]: (ecto_sql 3.4.5) lib/ecto/adapters/sql.ex:526: Ecto.Adapters.SQL.execute/5
Mar 08 16:14:39 procyon pleroma[1777730]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:192: Ecto.Repo.Queryable.execute/4
Mar 08 16:14:39 procyon pleroma[1777730]: (ecto 3.4.6) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
Mar 08 16:14:39 procyon pleroma[1777730]: (pleroma 2.3.0-1-gb221d77a) lib/pleroma/pagination.ex:40: Pleroma.Pagination.fetch_paginated/4
Mar 08 16:14:39 procyon pleroma[1777730]: (pleroma 2.3.0-1-gb221d77a) lib/pleroma/web/mastodon_api/controllers/notification_controller.ex:59: Pleroma.Web.MastodonAPI.NotificationController.index/2
Mar 08 16:14:39 procyon pleroma[1777730]: (pleroma 2.3.0-1-gb221d77a) lib/pleroma/web/mastodon_api/controllers/notification_controller.ex:5: Pleroma.Web.MastodonAPI.NotificationController.action/2
Mar 08 16:14:39 procyon pleroma[1777730]: (pleroma 2.3.0-1-gb221d77a) lib/pleroma/web/mastodon_api/controllers/notification_controller.ex:5: Pleroma.Web.MastodonAPI.NotificationController.phoenix_controller_pipeline/2
This manifests as being unable to load notifications on the frontend, but receiving and sending statuses otherwise appears to be working correctly.
I should be able to restore from my backups, but I wonder if this is fixable without having to do that.
Edited by Josh Holland