Unknown error knocks out ecto
So this has happened a few times now, but at least I've figured out what's causing it
something is happening that knocks out one of oban's workers, which in turn knocks out ecto and makes the instance return 500 errors for a short while until ecto picks itself back up. I don't think the first error in the log is related, but I left it in nonetheless
ec 12 17:33:27 pleroma mix[27175]: Server: ihatebeinga.live:80 (http)
Dec 12 17:33:27 pleroma mix[27175]: Request: POST /inbox
Dec 12 17:33:27 pleroma mix[27175]: ** (exit) an exception was raised:
Dec 12 17:33:27 pleroma mix[27175]: ** (FunctionClauseError) no function clause matching in Plug.Conn.resp/3
Dec 12 17:33:27 pleroma mix[27175]: (plug) lib/plug/conn.ex:577: Plug.Conn.resp(%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{digest: "SHA-256=8igTXwqfL02/MiTMJDkLx71wemF5pwtBgiEFgJeerB4=", kind: :error, layout: false, locale: "en", reason: %MatchError{term: {:error, {:error, {:options, {:socket_options, [socks5_transport: :hackney_ssl, socks5_resolve: :undefined, socks5_pass: :undefined, socks5_user: :undefined, socks5_port: 1080, socks5_host: {a, b, c, d}, insecure: false, ssl_options: [partial_chain: &:hackney_connect.partial_chain/1, versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"]], packet_size: 0, packet: 0, header: 0, active: false, mode: :binary]}}}}}, stack: [{Pleroma.Web.ActivityPub.Transmogrifier, :fix_addressing, 1, [file: 'lib/pleroma/web/activity_pub/transmogrifier.ex', line: 137]}, {Pleroma.Web.ActivityPub.Transmogrifier, :handle_incoming, 2, [file: 'lib/pleroma/web/activity_pub/transmogrifier.ex', line: 405]}, {Pleroma.Object.Fetcher, :fetch_object_from_id, 2, [file: 'lib/pleroma/object/fetcher.ex', line: 71]}, {Pleroma.Web.ActivityPub.ActivityPubController, :inbox, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 264]}, {Pleroma.Web.ActivityPub.ActivityPubController, :action, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 5]}, {Pleroma.Web.ActivityPub.ActivityPubController, :phoenix_controller_pipeline, 2, [file: 'lib/pleroma/web/activity_pub/activity_pub_controller.ex', line: 5]}, {Pleroma.Web.Endpoint, :instrument, 4, [file: 'lib/pleroma/web/endpoint.ex', line: 5]}, {Phoenix.Router, :__call__, 2, [file: 'lib/phoenix/router.ex', line: 288]}], valid_signature: false}, before_send: [#Function<0.61458113/1 in Pleroma.Web.Endpoint.PipelineInstrumenter.call/2>, #Function<1.16531471/1 in Plug.Logger.call/2>], body_params: %{"@context" => ["https://www.w3.org/ns/activitystreams", %{"atomUri" => "ostatus:atomUri", "blurhash" => "toot:blurhash", "conversation" => "ostatus:conversation", "focalPoint" => %{"@container" => "@list", "@id" => "toot:focalPoint"}, "inReplyToAtomUri" => "ostatus:inReplyToAtomUri", "ostatus" => "http://ostatus.org#", "sensitive" => "as:sensitive", "toot" => "http://joinmastodon.org/ns#", "votersCount" => "toot:votersCount"}], "actor" => "https://sleeping.town/users/BrutalSlab", "cc" => ["https://www.w3.org/ns/activitystreams#Public"], "id" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003/activity", "object" => %{"atomUri" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003", "attachment" => [%{"blurhash" => "U76%+woK0zkB$*R+NHs:IpWVxZjtWCoeoeR*", "mediaType" => "video/mp4", "name" => nil, "type" => "Document", "url" => "https://pool.jortage.com/sleepingtown/media_attachments/files/002/477/621/original/f267373140faba49.mp4"}], "attributedTo" => "https://sleeping.town/users/BrutalSlab", "cc" => ["https://www.w3.org/ns/activitystreams#Public"], "content" => "<p>Wishbone: The Legend of Sleepy Hollow - VHS Review - brutalmoose<br />Uploaded October 30 2017</p><p><a href=\"https://youtube.com/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">youtube.com/watch?v=ziL-iSVYmt</span><span class=\"invisible\">s&t=910</span></a><br /><a href=\"https://invidio.us/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">invidio.us/watch?v=ziL-iSVYmts</span><span class=\"invisible\">&t=910</span></a></p>", "contentMap" => %{"en" => "<p>Wishbone: The Legend of Sleepy Hollow - VHS Review - brutalmoose<br />Uploaded October 30 2017</p><p><a href=\"https://youtube.com/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">youtube.com/watch?v=ziL-iSVYmt</span><span class=\"invisible\">s&t=910</span></a><br /><a href=\"https://invidio.us/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">invidio.us/watch?v=ziL-iSVYmts</span><span class=\"invisible\">&t=910</span></a></p>"}, "conversation" => "tag:sleeping.town,2019-12-12:objectId=13621055:objectType=Conversation", "id" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003", "inReplyTo" => nil, "inReplyToAtomUri" => nil, "published" => "2019-12-12T17:30:09Z", "replies" => %{"first" => %{"items" => [], "next" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003/replies?only_other_accounts=true&page=true", "partOf" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003/replies", "type" => "CollectionPage"}, "id" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003/replies", "type" => "Collection"}, "sensitive" => false, "summary" => nil, "tag" => [], "to" => ["https://sleeping.town/users/BrutalSlab/followers"], "type" => "Note", "url" => "https://sleeping.town/@BrutalSlab/103295995673285003"}, "published" => "2019-12-12T17:30:09Z", "to" => ["https://sleeping.town/users/BrutalSlab/followers"], "type" => "Create"}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "ihatebeinga.live", method: "POST", owner: #PID<0.30144.28>, params: %{"@context" => ["https://www.w3.org/ns/activitystreams", %{"atomUri" => "ostatus:atomUri", "blurhash" => "toot:blurhash", "conversation" => "ostatus:conversation", "focalPoint" => %{"@container" => "@list", "@id" => "toot:focalPoint"}, "inReplyToAtomUri" => "ostatus:inReplyToAtomUri", "ostatus" => "http://ostatus.org#", "sensitive" => "as:sensitive", "toot" => "http://joinmastodon.org/ns#", "votersCount" => "toot:votersCount"}], "actor" => "https://sleeping.town/users/BrutalSlab", "cc" => ["https://www.w3.org/ns/activitystreams#Public"], "id" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003/activity", "object" => %{"atomUri" => "https://sleeping.town/users/BrutalSlab/statuses/103295995673285003", "attachment" => [%{"blurhash" => "U76%+woK0zkB$*R+NHs:IpWVxZjtWCoeoeR*", "mediaType" => "video/mp4", "name" => nil, "type" => "Document", "url" => "https://pool.jortage.com/sleepingtown/media_attachments/files/002/477/621/original/f267373140faba49.mp4"}], "attributedTo" => "https://sleeping.town/users/BrutalSlab", "cc" => ["https://www.w3.org/ns/activitystreams#Public"], "content" => "<p>Wishbone: The Legend of Sleepy Hollow - VHS Review - brutalmoose<br />Uploaded October 30 2017</p><p><a href=\"https://youtube.com/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">youtube.com/watch?v=ziL-iSVYmt</span><span class=\"invisible\">s&t=910</span></a><br /><a href=\"https://invidio.us/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">invidio.us/watch?v=ziL-iSVYmts</span><span class=\"invisible\">&t=910</span></a></p>", "contentMap" => %{"en" => "<p>Wishbone: The Legend of Sleepy Hollow - VHS Review - brutalmoose<br />Uploaded October 30 2017</p><p><a href=\"https://youtube.com/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">youtube.com/watch?v=ziL-iSVYmt</span><span class=\"invisible\">s&t=910</span></a><br /><a href=\"https://invidio.us/watch?v=ziL-iSVYmts&t=910\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">invidio.us/watch?v=ziL-iSVYmts</span><span class=\"invisible\">&t=910</span></a></p>"}, "conversation" => "t (truncated)
Dec 12 17:33:37 pleroma mix[27175]: 17:33:37.789 [error] Task #PID<0.28321.28> started from Oban.Queue.FederatorOutgoing.Foreman terminating
Dec 12 17:33:37 pleroma mix[27175]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1300ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more informationI'll continue to investigate as and when it happens, and there's not an awful lot to go on, but i'll log it here now as a "it happens" thing