random crashes
Linux saturne 4.19.0-6-cloud-amd64 #1 (closed) SMP Debian 4.19.67-2 (2019-08-28) x86_64 GNU/Linux Using a 2GB RAM VPS, debian 10 buster, pleroma 1.0.6
pleroma instance crashes randomly, about once per day. I cannot reproduce it. I noticed that after a crash, postgreSQL job keeps running, and "/home/pleroma/erts-10.3.5.2/bin/epmd -daemon" task is still running.
The problem seems to be related to one (or both) of this jobs:
/home/pleroma/releases/1.0.6/../../erts-10.3.5.2/bin/run_erl /home/pleroma/erts-10.3.5.2/bin/beam.smp
After a crash, both those two jobs doesnt exist anymore. file "erl_crash.dump" is created: I can upload it somewhere if needed.
Here are my logs at 21:07, when crash happened:
Sep 24 21:06:50 saturne pleroma: [error] Couldn't fetch ""https://mastodon.social/users/knuxfanwin8/statuses/102407989994603591"", error: nil Sep 24 21:06:50 saturne pleroma: [error] Task #PID<0.1116.1> started from PleromaJobQueue.Worker terminating#012** (MatchError) no match of right hand side value: nil#012 (pleroma) lib/pleroma/web/activity_pub/utils.ex:193: Pleroma.Web.ActivityPub.Utils.lazy_put_activity_defaults/2#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:124: Pleroma.Web.ActivityPub.ActivityPub.insert/3#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:265: Pleroma.Web.ActivityPub.ActivityPub.create/2#012 (pleroma) lib/pleroma/object/fetcher.ex:42: Pleroma.Object.Fetcher.fetch_object_from_id/2#012 (pleroma) lib/pleroma/object/fetcher.ex:73: Pleroma.Object.Fetcher.fetch_object_from_id!/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:752: Pleroma.Web.ActivityPub.Transmogrifier.get_obj_helper/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:177: Pleroma.Web.ActivityPub.Transmogrifier.fix_in_reply_to/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:33: Pleroma.Web.ActivityPub.Transmogrifier.fix_object/2#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#012 Args: [] Sep 24 21:06:55 saturne pleroma: [error] Task #PID<0.1659.1> started from PleromaJobQueue.Worker terminating#012** (MatchError) no match of right hand side value: nil#012 (pleroma) lib/pleroma/web/activity_pub/utils.ex:193: Pleroma.Web.ActivityPub.Utils.lazy_put_activity_defaults/2#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:124: Pleroma.Web.ActivityPub.ActivityPub.insert/3#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:265: Pleroma.Web.ActivityPub.ActivityPub.create/2#012 (pleroma) lib/pleroma/object/fetcher.ex:42: Pleroma.Object.Fetcher.fetch_object_from_id/2#012 (pleroma) lib/pleroma/object/fetcher.ex:73: Pleroma.Object.Fetcher.fetch_object_from_id!/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:752: Pleroma.Web.ActivityPub.Transmogrifier.get_obj_helper/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:177: Pleroma.Web.ActivityPub.Transmogrifier.fix_in_reply_to/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:33: Pleroma.Web.ActivityPub.Transmogrifier.fix_object/2#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#012 Args: [] Sep 24 21:06:56 saturne pleroma: [error] Couldn't fetch ""https://mast.eu.org/users/BrunoBellamy/statuses/102173873323637090"", error: nil Sep 24 21:06:56 saturne pleroma: [error] Task #PID<0.1793.1> started from PleromaJobQueue.Worker terminating#012** (MatchError) no match of right hand side value: nil#012 (pleroma) lib/pleroma/web/activity_pub/utils.ex:193: Pleroma.Web.ActivityPub.Utils.lazy_put_activity_defaults/2#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:124: Pleroma.Web.ActivityPub.ActivityPub.insert/3#012 (pleroma) lib/pleroma/web/activity_pub/activity_pub.ex:265: Pleroma.Web.ActivityPub.ActivityPub.create/2#012 (pleroma) lib/pleroma/object/fetcher.ex:42: Pleroma.Object.Fetcher.fetch_object_from_id/2#012 (pleroma) lib/pleroma/object/fetcher.ex:73: Pleroma.Object.Fetcher.fetch_object_from_id!/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:752: Pleroma.Web.ActivityPub.Transmogrifier.get_obj_helper/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:177: Pleroma.Web.ActivityPub.Transmogrifier.fix_in_reply_to/2#012 (pleroma) lib/pleroma/web/activity_pub/transmogrifier.ex:33: Pleroma.Web.ActivityPub.Transmogrifier.fix_object/2#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#012 Args: []
looks like database is having troubles:
Sep 24 21:07:02 saturne pleroma: [error] Task #PID<0.2568.1> started from PleromaJobQueue.Worker terminating#012** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 813ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information#012 (ecto_sql) lib/ecto/adapters/sql.ex:618: Ecto.Adapters.SQL.raise_sql_call_error/1#012 (ecto_sql) lib/ecto/adapters/sql.ex:551: Ecto.Adapters.SQL.execute/5#012 (ecto) lib/ecto/repo/queryable.ex:153: Ecto.Repo.Queryable.execute/4#012 (ecto) lib/ecto/repo/queryable.ex:18: Ecto.Repo.Queryable.all/3#012 (ecto) lib/ecto/repo/queryable.ex:67: Ecto.Repo.Queryable.one/3#012 (pleroma) lib/pleroma/web/federator/federator.ex:116: Pleroma.Web.Federator.perform/2#012 (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2#012 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3#012Function: #Function<2.25193041/0 in PleromaJobQueue.Worker.maybe_start_job/4>#12 (closed) Args: []
thank for your help, cheers, Ranma