Commit a90ea8ba authored by Ivan Tashkinov's avatar Ivan Tashkinov
Browse files

[#1149] Addressed code review comments (code style, jobs pruning etc.).

parent e890ea7e
......@@ -19,7 +19,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mastodon API: Unsubscribe followers when they unfollow a user
- AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
- Improve digest email template
- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) with [Oban](https://github.com/sorentwo/oban)
- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings).
- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler
### Fixed
......
......@@ -470,7 +470,7 @@
config :pleroma, Oban,
repo: Pleroma.Repo,
verbose: false,
prune: {:maxage, 60 * 60 * 24 * 7},
prune: {:maxlen, 1500},
queues: [
activity_expiration: 10,
federator_incoming: 50,
......
......@@ -65,6 +65,8 @@
queues: false,
prune: :disabled
config :pleroma, Pleroma.Scheduler, jobs: []
config :pleroma, Pleroma.ScheduledActivity,
daily_user_limit: 2,
total_user_limit: 3,
......
......@@ -404,20 +404,29 @@ curl "http://localhost:4000/api/pleroma/admin/invite_token?admin_token=somerando
[Oban](https://github.com/sorentwo/oban) asynchronous job processor configuration.
Configuration options described in [Oban readme](https://github.com/sorentwo/oban#usage):
* `repo` - app's Ecto repo (`Pleroma.Repo`)
* `verbose` - logs verbosity
* `prune` - non-retryable jobs [pruning settings](https://github.com/sorentwo/oban#pruning) (`:disabled` / `{:maxlen, value}` / `{:maxage, value}`)
* `queues` - job queues (see below)
Pleroma has the following queues:
* `activity_expiration` - Activity expiration
* `federator_outgoing` - Outgoing federation
* `federator_incoming` - Incoming federation
* `mailer` - Email sender, see [`Pleroma.Emails.Mailer`](#pleroma-emails-mailer)
* `mailer` - Email sender, see [`Pleroma.Emails.Mailer`](#pleromaemailsmailer)
* `transmogrifier` - Transmogrifier
* `web_push` - Web push notifications
* `scheduled_activities` - Scheduled activities, see [`Pleroma.ScheduledActivities`](#pleromascheduledactivity)
* `scheduled_activities` - Scheduled activities, see [`Pleroma.ScheduledActivity`](#pleromascheduledactivity)
Example:
```elixir
config :pleroma, Oban,
repo: Pleroma.Repo,
verbose: false,
prune: {:maxlen, 1500},
queues: [
federator_incoming: 50,
federator_outgoing: 50
......@@ -426,12 +435,37 @@ config :pleroma, Oban,
This config contains two queues: `federator_incoming` and `federator_outgoing`. Both have the number of max concurrent jobs set to `50`.
### Migrating `pleroma_job_queue` settings
`config :pleroma_job_queue, :queues` is replaced by `config :pleroma, Oban, :queues` and uses the same format (keys are queues' names, values are max concurrent jobs numbers).
### Note on running with PostgreSQL in silent mode
If you are running PostgreSQL in [`silent_mode`](https://postgresqlco.nf/en/doc/param/silent_mode?version=9.1), it's advised to set [`log_destination`](https://postgresqlco.nf/en/doc/param/log_destination?version=9.1) to `syslog`,
otherwise `postmaster.log` file may grow because of "you don't own a lock of type ShareLock" warnings (see https://github.com/sorentwo/oban/issues/52).
## :workers
Includes custom worker options not interpretable directly by `Oban`.
* `retries` — keyword lists where keys are `Oban` queues (see above) and values are numbers of max attempts for failed jobs.
Example:
```elixir
config :pleroma, :workers,
retries: [
federator_incoming: 5,
federator_outgoing: 5
]
```
### Migrating `Pleroma.Web.Federator.RetryQueue` settings
* `max_retries` is replaced with `config :pleroma, :workers, retries: [federator_outgoing: 5]`
* `enabled: false` corresponds to `config :pleroma, :workers, retries: [federator_outgoing: 1]`
* deprecated options: `max_jobs`, `initial_timeout`
## Pleroma.Web.Metadata
* `providers`: a list of metadata providers to enable. Providers available:
* Pleroma.Web.Metadata.Providers.OpenGraph
......@@ -491,6 +525,24 @@ config :auto_linker,
]
```
## Pleroma.Scheduler
Configuration for [Quantum](https://github.com/quantum-elixir/quantum-core) jobs scheduler.
See [Quantum readme](https://github.com/quantum-elixir/quantum-core#usage) for the list of supported options.
Example:
```elixir
config :pleroma, Pleroma.Scheduler,
global: true,
overlap: true,
timezone: :utc,
jobs: [{"0 */6 * * * *", {Pleroma.Web.Websub, :refresh_subscriptions, []}}]
```
The above example defines a single job which invokes `Pleroma.Web.Websub.refresh_subscriptions()` every 6 hours ("0 */6 * * * *", [crontab format](https://en.wikipedia.org/wiki/Cron)).
## Pleroma.ScheduledActivity
* `daily_user_limit`: the number of scheduled activities a user is allowed to create in a single day (Default: `25`)
......
......@@ -9,13 +9,13 @@ defmodule Pleroma.ActivityExpirationWorker do
alias Pleroma.Repo
alias Pleroma.User
alias Pleroma.Web.CommonAPI
alias Pleroma.Workers.BackgroundWorker
alias Pleroma.Workers.ActivityExpirationWorker
require Logger
use GenServer
import Ecto.Query
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
@schedule_interval :timer.minutes(1)
......@@ -57,7 +57,7 @@ def handle_info(:perform, state) do
"op" => "activity_expiration",
"activity_expiration_id" => expiration.id
}
|> BackgroundWorker.new(worker_args(:activity_expiration))
|> ActivityExpirationWorker.new(worker_args(:activity_expiration))
|> Repo.insert()
end)
......
......@@ -43,7 +43,7 @@ def start(_type, _args) do
hackney_pool_children() ++
[
Pleroma.Stats,
{Oban, Application.get_env(:pleroma, Oban)},
{Oban, Pleroma.Config.get(Oban)},
%{
id: :web_push_init,
start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
......
......@@ -4,11 +4,11 @@
defmodule Pleroma.DigestEmailWorker do
alias Pleroma.Repo
alias Pleroma.Workers.Mailer, as: MailerWorker
alias Pleroma.Workers.MailerWorker
import Ecto.Query
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def perform do
config = Pleroma.Config.get([:email_notifications, :digest])
......
......@@ -10,7 +10,7 @@ defmodule Pleroma.Emails.Mailer do
"""
alias Pleroma.Repo
alias Pleroma.Workers.Mailer, as: MailerWorker
alias Pleroma.Workers.MailerWorker
alias Swoosh.DeliveryError
@otp_app :pleroma
......@@ -19,7 +19,7 @@ defmodule Pleroma.Emails.Mailer do
@spec enabled?() :: boolean()
def enabled?, do: Pleroma.Config.get([__MODULE__, :enabled])
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
@doc "add email to queue"
def deliver_async(email, config \\ []) do
......
......@@ -18,7 +18,7 @@ defmodule Pleroma.ScheduledActivityWorker do
@schedule_interval :timer.minutes(1)
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def start_link(_) do
GenServer.start_link(__MODULE__, nil)
......
......@@ -41,7 +41,7 @@ defmodule Pleroma.User do
@strict_local_nickname_regex ~r/^[a-zA-Z\d]+$/
@extended_local_nickname_regex ~r/^[a-zA-Z\d_-]+$/
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
schema "users" do
field(:bio, :string)
......
......@@ -26,7 +26,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
require Logger
require Pleroma.Constants
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
# For Announce activities, we filter the recipients based on following status for any actors
# that match actual users. See issue #164 for more information about why this is necessary.
......
......@@ -18,7 +18,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy do
recv_timeout: 10_000
]
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def perform(:prefetch, url) do
Logger.info("Prefetching #{inspect(url)}")
......
......@@ -85,7 +85,7 @@ def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = pa
end
def publish_one(%{actor_id: actor_id} = params) do
actor = User.get_by_id(actor_id)
actor = User.get_cached_by_id(actor_id)
params
|> Map.delete(:actor_id)
......
......@@ -15,14 +15,14 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
alias Pleroma.Web.ActivityPub.Utils
alias Pleroma.Web.ActivityPub.Visibility
alias Pleroma.Web.Federator
alias Pleroma.Workers.Transmogrifier, as: TransmogrifierWorker
alias Pleroma.Workers.TransmogrifierWorker
import Ecto.Query
require Logger
require Pleroma.Constants
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
@doc """
Modifies an incoming AP object (mastodon format) to our internal format.
......
......@@ -12,13 +12,13 @@ defmodule Pleroma.Web.Federator do
alias Pleroma.Web.Federator.Publisher
alias Pleroma.Web.OStatus
alias Pleroma.Web.Websub
alias Pleroma.Workers.Publisher, as: PublisherWorker
alias Pleroma.Workers.Receiver, as: ReceiverWorker
alias Pleroma.Workers.Subscriber, as: SubscriberWorker
alias Pleroma.Workers.PublisherWorker
alias Pleroma.Workers.ReceiverWorker
alias Pleroma.Workers.SubscriberWorker
require Logger
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def init do
# To do: consider removing this call in favor of scheduled execution (`quantum`-based)
......
......@@ -6,7 +6,7 @@ defmodule Pleroma.Web.Federator.Publisher do
alias Pleroma.Activity
alias Pleroma.Config
alias Pleroma.User
alias Pleroma.Workers.Publisher, as: PublisherWorker
alias Pleroma.Workers.PublisherWorker
require Logger
......@@ -31,12 +31,7 @@ defmodule Pleroma.Web.Federator.Publisher do
"""
@spec enqueue_one(module(), Map.t()) :: :ok
def enqueue_one(module, %{} = params) do
worker_args =
if max_attempts = Pleroma.Config.get([:workers, :retries, :federator_outgoing]) do
[max_attempts: max_attempts]
else
[]
end
worker_args = Pleroma.Workers.WorkerHelper.worker_args(:federator_outgoing)
%{"op" => "publish_one", "module" => to_string(module), "params" => params}
|> PublisherWorker.new(worker_args)
......
......@@ -20,7 +20,7 @@ defmodule Pleroma.Web.OAuth.Token.CleanWorker do
alias Pleroma.Web.OAuth.Token
alias Pleroma.Workers.BackgroundWorker
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def start_link(_), do: GenServer.start_link(__MODULE__, %{})
......
......@@ -4,11 +4,11 @@
defmodule Pleroma.Web.Push do
alias Pleroma.Repo
alias Pleroma.Workers.WebPusher
alias Pleroma.Workers.WebPusherWorker
require Logger
defdelegate worker_args(queue), to: Pleroma.Workers.Helper
import Pleroma.Workers.WorkerHelper, only: [worker_args: 1]
def init do
unless enabled() do
......@@ -36,7 +36,7 @@ def enabled do
def send(notification) do
%{"op" => "web_push", "notification_id" => notification.id}
|> WebPusher.new(worker_args(:web_push))
|> WebPusherWorker.new(worker_args(:web_push))
|> Repo.insert()
end
end
......@@ -171,7 +171,7 @@ def publish_one(%{recipient: url, feed: feed} = params) when is_binary(url) do
end
def publish_one(%{recipient_id: recipient_id} = params) do
recipient = User.get_by_id(recipient_id)
recipient = User.get_cached_by_id(recipient_id)
params
|> Map.delete(:recipient_id)
......
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Workers.ActivityExpirationWorker do
# Note: `max_attempts` is intended to be overridden in `new/2` call
use Oban.Worker,
queue: "activity_expiration",
max_attempts: 1
@impl Oban.Worker
def perform(
%{
"op" => "activity_expiration",
"activity_expiration_id" => activity_expiration_id
},
_job
) do
Pleroma.ActivityExpirationWorker.perform(:execute, activity_expiration_id)
end
end
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment