random daemon crashes
Using pleroma 1.0.0-7-g9504544d (OTP release) psql (PostgreSQL) 9.6.13 Debian 9 stretch VPS (4.9.88 #1 (closed) SMP Thu May 24 18:16:43 CEST 2018 x86_64 GNU/Linux) 2GB RAM
daemon crashes randomly, about 1-2 times every 24h. It does not depend on an action on my part, because I am alone on my instance, and this happens sometimes when I do not use it.
When I enabled the relay feature (relay follow https://relay.mastodon.host/actor), crashes were much more frequent. Now that I disabled it, less crashes.
last lines of postgresql log before crash:
2019-07-03 20:40:21.350 CEST [47789] pleroma@pleroma INSTRUCTION : INSERT INTO "objects" ("data","inserted_at","updated_at") VALUES ($1,$2,$3) RETURNING "id" 2019-07-03 20:40:23.155 CEST [47794] pleroma@pleroma ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « objects_unique_apid_index » 2019-07-03 20:40:23.155 CEST [47794] pleroma@pleroma DÉTAIL: La clé « ((data ->> 'id'::text))=(https://mastodon.ar.al/users/aral/statuses/101069121495698485) » existe déjà. 2019-07-03 20:40:23.155 CEST [47794] pleroma@pleroma INSTRUCTION : INSERT INTO "objects" ("data","inserted_at","updated_at") VALUES ($1,$2,$3) RETURNING "id" 2019-07-03 20:40:24.437 CEST [47785] pleroma@pleroma ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « objects_unique_apid_index » 2019-07-03 20:40:24.437 CEST [47785] pleroma@pleroma DÉTAIL: La clé « ((data ->> 'id'::text))=(tag:mastodon.social,2018-12-31:objectId=75274730:objectType=Conversation) » existe déjà. 2019-07-03 20:40:24.437 CEST [47785] pleroma@pleroma INSTRUCTION : INSERT INTO "objects" ("data","inserted_at","updated_at") VALUES ($1,$2,$3) RETURNING "id"
last lines of syslog before crash
Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2266.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: [] Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2267.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: [] Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2268.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: [] Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2269.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: [] Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2270.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: [] Jul 3 20:40:29 saturne pleroma: [error] Task #PID<0.2271.1> started from PleromaJobQueue.Worker terminating#012** (ArgumentError) argument error#012 (stdlib) :ets.lookup(:cachex_overseer_table, :user_cache)#12 (closed) (cachex) lib/cachex/services/overseer.ex:89: Cachex.Services.Overseer.retrieve/1#012 (cachex) lib/cachex.ex:665: Cachex.fetch/4#012 (cachex) lib/cachex.ex:1: Cachex.fetch!/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:140: Pleroma.Web.Federator.ap_enabled_actor/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:103: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: []
sometimes, I get different error messages:
Jul 3 14:33:29 saturne pleroma: [error] Task #PID<0.1678.1> started from PleromaJobQueue.Worker terminating#012** (Cachex.ExecutionError) connection not available and request was dropped from queue after 177ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information#012 (cachex) lib/cachex.ex:1447: Cachex.unwrap_unsafe/1#012 (pleroma) lib/pleroma/object/fetcher.ex:26: Pleroma.Object.Fetcher.fetch_object_from_id/1#012 (pleroma) lib/pleroma/object/fetcher.ex:67: Pleroma.Object.Fetcher.fetch_object_from_id!/1#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:723: Pleroma.Web.ActivityPub.Transmogrifier.get_obj_helper/1#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:185: Pleroma.Web.ActivityPub.Transmogrifier.fix_in_reply_to/1#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:31: Pleroma.Web.ActivityPub.Transmogrifier.fix_object/1#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:430: Pleroma.Web.ActivityPub.Transmogrifier.handle_incoming/1#012 (pleroma) lib/pleroma/web/federator/federator.ex:106: Pleroma.Web.Federator.perform/2#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: []
I tried to prune database, and VACUUM ANALYZE, it doesnt help.
any help will be much welcomed ! cheers, Julien