Merge branch 'develop' into 'domain-block-precedence'

# Conflicts:
#   lib/pleroma/user.ex

.formatter.exs

 [
-  inputs: ["mix.exs", "{config,lib,test}/**/*.{ex,exs}", "priv/repo/migrations/*.exs"]
+  inputs: ["mix.exs", "{config,lib,test}/**/*.{ex,exs}", "priv/repo/migrations/*.exs", "priv/scrubbers/*.ex"]
 ]

.gitlab-ci.yml

@@ -31,6 +31,7 @@ build:
 
 benchmark:
   stage: benchmark
+  when: manual
   variables:
     MIX_ENV: benchmark
   services:
@@ -55,6 +56,19 @@ unit-testing:
     - mix ecto.migrate
     - mix coveralls --preload-modules
 
+federated-testing:
+  stage: test
+  services:
+  - name: minibikini/postgres-with-rum:12
+    alias: postgres
+    command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"]
+  script:
+    - mix deps.get
+    - mix ecto.create
+    - mix ecto.migrate
+    - epmd -daemon
+    - mix test --trace --only federated
+
 unit-testing-rum:
   stage: test
   services:

CHANGELOG.md

@@ -20,6 +20,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - OStatus: Extract RSS functionality
 - Deprecated `User.Info` embedded schema (fields moved to `User`)
 - Store status data inside Flag activity
+- Deprecated (reorganized as `UserRelationship` entity) User fields with user AP IDs (`blocks`, `mutes`, `muted_reblogs`, `muted_notifications`, `subscribers`).
 <details>
   <summary>API Changes</summary>
 
@@ -35,9 +36,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: `pleroma.thread_muted` to the Status entity
 - Mastodon API: Mark the direct conversation as read for the author when they send a new direct message
 - Mastodon API, streaming: Add `pleroma.direct_conversation_id` to the `conversation` stream event payload.
+- Admin API: Render whole status in grouped reports
+- Mastodon API: User timelines will now respect blocks, unless you are getting the user timeline of somebody you blocked (which would be empty otherwise).
 </details>
 
 ### Added
+- `:chat_limit` option to limit chat characters.
 - Refreshing poll results for remote polls
 - Authentication: Added rate limit for password-authorized actions / login existence checks
 - Static Frontend: Add the ability to render user profiles and notices server-side without requiring JS app.
@@ -45,6 +49,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mix task to list all users (`mix pleroma.user list`)
 - Support for `X-Forwarded-For` and similar HTTP headers which used by reverse proxies to pass a real user IP address to the backend. Must not be enabled unless your instance is behind at least one reverse proxy (such as Nginx, Apache HTTPD or Varnish Cache).
 - MRF: New module which handles incoming posts based on their age. By default, all incoming posts that are older than 2 days will be unlisted and not shown to their followers.
+- User notification settings: Add `privacy_option` option.
+- User settings: Add _This account is a_ option.
+- OAuth: admin scopes support (relevant setting: `[:auth, :enforce_oauth_admin_scope_usage]`).
 <details>
   <summary>API Changes</summary>
 
@@ -66,22 +73,30 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
   - `DELETE /api/pleroma/admin/users` (`nickname` query param or `nickname` sent in JSON body) is deprecated in favor of: `DELETE /api/pleroma/admin/users` (`nicknames` query array param or `nicknames` sent in JSON body)
 - Admin API: Add `GET /api/pleroma/admin/relay` endpoint - lists all followed relays
 - Pleroma API: `POST /api/v1/pleroma/conversations/read` to mark all conversations as read
+- ActivityPub: Support `Move` activities
 - Mastodon API: Add `/api/v1/markers` for managing timeline read markers
 - Mastodon API: Add the `recipients` parameter to `GET /api/v1/conversations`
 - Configuration: `feed` option for user atom feed.
 - Pleroma API: Add Emoji reactions
 - Admin API: Add `/api/pleroma/admin/instances/:instance/statuses` - lists all statuses from a given instance
 - Admin API: `PATCH /api/pleroma/users/confirm_email` to confirm email for multiple users, `PATCH /api/pleroma/users/resend_confirmation_email` to resend confirmation email for multiple users
+- ActivityPub: Configurable `type` field of the actors.
+- Mastodon API: `/api/v1/accounts/:id` has `source/pleroma/actor_type` field.
+- Mastodon API: `/api/v1/update_credentials` accepts `actor_type` field.
 </details>
 
 ### Fixed
 - Report emails now include functional links to profiles of remote user accounts
 - Not being able to log in to some third-party apps when logged in to MastoFE
+- MRF: `Delete` activities being exempt from MRF policies
+- OTP releases: Not being able to configure OAuth expired token cleanup interval
+- OTP releases: Not being able to configure HTML sanitization policy
 <details>
   <summary>API Changes</summary>
 
 - Mastodon API: Fix private and direct statuses not being filtered out from the public timeline for an authenticated user (`GET /api/v1/timelines/public`)
 - Mastodon API: Inability to get some local users by nickname in `/api/v1/accounts/:id_or_nickname`
+- AdminAPI: If some status received reports both in the "new" format and "old" format it was considered reports on two different statuses (in the context of grouped reports)
 - Admin API: Error when trying to update reports in the "old" format
 </details>
 

benchmarks/load_testing/fetcher.ex

@@ -95,7 +95,36 @@ def query_timelines(user) do
           for: user,
           as: :activity
         })
-      end
+      end,
+      "Rendering favorites timeline" => fn ->
+        conn = Phoenix.ConnTest.build_conn(:get, "http://localhost:4001/api/v1/favourites", nil)
+        Pleroma.Web.MastodonAPI.StatusController.favourites(
+          %Plug.Conn{conn |
+                     assigns: %{user: user},
+                     query_params:  %{"limit" => "0"},
+                     body_params: %{},
+                     cookies: %{},
+                     params: %{},
+                     path_params: %{},
+                     private: %{
+                       Pleroma.Web.Router => {[], %{}},
+                       phoenix_router: Pleroma.Web.Router,
+                       phoenix_action: :favourites,
+                       phoenix_controller: Pleroma.Web.MastodonAPI.StatusController,
+                       phoenix_endpoint: Pleroma.Web.Endpoint,
+                       phoenix_format: "json",
+                       phoenix_layout: {Pleroma.Web.LayoutView, "app.html"},
+                       phoenix_recycled: true,
+
+                       phoenix_view: Pleroma.Web.MastodonAPI.StatusView,
+                       plug_session: %{"user_id" => user.id},
+                       plug_session_fetch: :done,
+                       plug_session_info: :write,
+                       plug_skip_csrf_protection: true
+                     }
+          },
+          %{})
+      end,
     })
   end
 

benchmarks/load_testing/generator.ex

@@ -2,6 +2,24 @@ defmodule Pleroma.LoadTesting.Generator do
   use Pleroma.LoadTesting.Helper
   alias Pleroma.Web.CommonAPI
 
+  def generate_like_activities(user, posts) do
+    count_likes = Kernel.trunc(length(posts) / 4)
+    IO.puts("Starting generating #{count_likes} like activities...")
+
+    {time, _} =
+      :timer.tc(fn ->
+        Task.async_stream(
+           Enum.take_random(posts, count_likes),
+          fn post -> {:ok, _, _} = CommonAPI.favorite(post.id, user) end,
+          max_concurrency: 10,
+          timeout: 30_000
+        )
+        |> Stream.run()
+      end)
+
+    IO.puts("Inserting like activities take #{to_sec(time)} sec.\n")
+  end
+
   def generate_users(opts) do
     IO.puts("Starting generating #{opts[:users_max]} users...")
     {time, _} = :timer.tc(fn -> do_generate_users(opts) end)
@@ -31,7 +49,6 @@ defp generate_user_data(i) do
       password_hash:
         "$pbkdf2-sha512$160000$bU.OSFI7H/yqWb5DPEqyjw$uKp/2rmXw12QqnRRTqTtuk2DTwZfF8VR4MYW2xMeIlqPR/UX1nT1CEKVUx2CowFMZ5JON8aDvURrZpJjSgqXrg",
       bio: "Tester Number #{i}",
-      info: %{},
       local: remote
     }
 

benchmarks/mix/tasks/pleroma/load_testing.ex

@@ -100,6 +100,10 @@ def run(args) do
 
     generate_remote_activities(user, remote_users)
 
+    generate_like_activities(
+      user, Pleroma.Repo.all(Pleroma.Activity.Queries.by_type("Create"))
+    )
+
     generate_dms(user, users, opts)
 
     {:ok, activity} = generate_long_thread(user, users, opts)

config/config.exs

@@ -225,6 +225,7 @@
   notify_email: "noreply@example.com",
   description: "A Pleroma instance, an alternative fediverse server",
   limit: 5_000,
+  chat_limit: 5_000,
   remote_limit: 100_000,
   upload_limit: 16_000_000,
   avatar_upload_limit: 2_000_000,
@@ -562,7 +563,10 @@
        base_path: "/oauth",
        providers: ueberauth_providers
 
-config :pleroma, :auth, oauth_consumer_strategies: oauth_consumer_strategies
+config :pleroma,
+       :auth,
+       enforce_oauth_admin_scope_usage: false,
+       oauth_consumer_strategies: oauth_consumer_strategies
 
 config :pleroma, Pleroma.Emails.Mailer, adapter: Swoosh.Adapters.Sendmail, enabled: false
 

config/description.exs

@@ -2094,6 +2094,15 @@
     type: :group,
     description: "Authentication / authorization settings",
     children: [
+      %{
+        key: :enforce_oauth_admin_scope_usage,
+        type: :boolean,
+        description:
+          "OAuth admin scope requirement toggle. " <>
+            "If `true`, admin actions explicitly demand admin OAuth scope(s) presence in OAuth token " <>
+            "(client app must support admin scopes). If `false` and token doesn't have admin scope(s)," <>
+            "`is_admin` user flag grants access to admin-specific actions."
+      },
       %{
         key: :auth_template,
         type: :string,

docs/API/admin_api.md

@@ -2,6 +2,13 @@
 
 Authentication is required and the user must be an admin.
 
+Configuration options:
+ 
+* `[:auth, :enforce_oauth_admin_scope_usage]` — OAuth admin scope requirement toggle.
+    If `true`, admin actions explicitly demand admin OAuth scope(s) presence in OAuth token (client app must support admin scopes).
+    If `false` and token doesn't have admin scope(s), `is_admin` user flag grants access to admin-specific actions.
+    Note that client app needs to explicitly support admin scopes and request them when obtaining auth token.
+
 ## `GET /api/pleroma/admin/users`
 
 ### List users

docs/API/differences_in_mastoapi_responses.md

@@ -57,6 +57,7 @@ Has these additional fields under the `pleroma` object:
 - `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`
 - `chat_token`: The token needed for Pleroma chat. Only returned in `verify_credentials`
 - `deactivated`: boolean, true when the user is deactivated
+- `allow_following_move`: boolean, true when the user allows automatically follow moved following accounts
 - `unread_conversation_count`: The count of unread conversations. Only returned to the account owner.
 
 ### Source
@@ -65,6 +66,8 @@ Has these additional fields under the `pleroma` object:
 
 - `show_role`: boolean, nullable, true when the user wants his role (e.g admin, moderator) to be shown
 - `no_rich_text` - boolean, nullable, true when html tags are stripped from all statuses requested from the API
+- `discoverable`: boolean, true when the user allows discovery of the account in search results and other services.
+- `actor_type`: string, the type of this account.
 
 ## Conversations
 
@@ -91,11 +94,18 @@ Has these additional fields under the `pleroma` object:
 
 - `is_seen`: true if the notification was read by the user
 
+### Move Notification
+
+The `type` value is `move`. Has an additional field:
+
+- `target`: new account
+
 ## GET `/api/v1/notifications`
 
 Accepts additional parameters:
 
 - `exclude_visibilities`: will exclude the notifications for activities with the given visibilities. The parameter accepts an array of visibility types (`public`, `unlisted`, `private`, `direct`). Usage example: `GET /api/v1/notifications?exclude_visibilities[]=direct&exclude_visibilities[]=private`.
+- `with_move`: boolean, when set to `true` will include Move notifications. `false` by default.
 
 ## POST `/api/v1/statuses`
 
@@ -136,7 +146,10 @@ Additional parameters can be added to the JSON body/Form data:
 - `default_scope` - the scope returned under `privacy` key in Source subentity
 - `pleroma_settings_store` - Opaque user settings to be saved on the backend.
 - `skip_thread_containment` - if true, skip filtering out broken threads
+- `allow_following_move` - if true, allows automatically follow moved following accounts
 - `pleroma_background_image` - sets the background image of the user.
+- `discoverable` - if true, discovery of this account in search results and other services is allowed.
+- `actor_type` - the type of this account.
 
 ### Pleroma Settings Store
 Pleroma has mechanism that allows frontends to save blobs of json for each user on the backend. This can be used to save frontend-specific settings for a user that the backend does not need to know about.

docs/API/pleroma_api.md

@@ -302,6 +302,7 @@ See [Admin-API](admin_api.md)
     * `follows`: BOOLEAN field, receives notifications from people the user follows
     * `remote`: BOOLEAN field, receives notifications from people on remote instances
     * `local`: BOOLEAN field, receives notifications from people on the local instance
+    * `privacy_option`: BOOLEAN field. When set to true, it removes the contents of a message from the push notification.
 * Response: JSON. Returns `{"status": "success"}` if the update was successful, otherwise returns `{"error": "error_msg"}`
 
 ## `/api/pleroma/healthcheck`

docs/administration/CLI_tasks/config.md

@@ -3,17 +3,26 @@
 !!! danger
     This is a Work In Progress, not usable just yet.
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl config` and in case of source installs it's
-`mix pleroma.config`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Transfer config from file to DB.
 
-```sh
-$PREFIX migrate_to_db
+```sh tab="OTP"
+ ./bin/pleroma_ctl config migrate_to_db
 ```
 
+```sh tab="From Source"
+mix pleroma.config migrate_to_db
+```
+
+
 ## Transfer config from DB to `config/env.exported_from_db.secret.exs`
 
-```sh
-$PREFIX migrate_from_db <env>
+```sh tab="OTP"
+ ./bin/pleroma_ctl config migrate_from_db <env>
 ```
+
+```sh tab="From Source"
+mix pleroma.config migrate_from_db <env>
+```
+

docs/administration/CLI_tasks/database.md

 # Database maintenance tasks
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl database` and in case of source installs it's `mix pleroma.database`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 !!! danger
     These mix tasks can take a long time to complete. Many of them were written to address specific database issues that happened because of bugs in migrations or other specific scenarios. Do not run these tasks "just in case" if everything is fine your instance.
@@ -9,8 +9,12 @@ Every command should be ran with a prefix, in case of OTP releases it is `./bin/
 
 Replaces embedded objects with references to them in the `objects` table. Only needs to be ran once if the instance was created before Pleroma 1.0.5. The reason why this is not a migration is because it could significantly increase the database size after being ran, however after this `VACUUM FULL` will be able to reclaim about 20% (really depends on what is in the database, your mileage may vary) of the db size before the migration.
 
-```sh
-$PREFIX remove_embedded_objects [<options>]
+```sh tab="OTP"
+./bin/pleroma_ctl database remove_embedded_objects [<options>]
+```
+
+```sh tab="From Source"
+mix pleroma.database remove_embedded_objects [<options>]
 ```
 
 ### Options
@@ -20,11 +24,15 @@ $PREFIX remove_embedded_objects [<options>]
 
 This will prune remote posts older than 90 days (configurable with [`config :pleroma, :instance, remote_post_retention_days`](../../configuration/cheatsheet.md#instance)) from the database, they will be refetched from source when accessed.
 
-!!! note
-    The disk space will only be reclaimed after `VACUUM FULL`
+!!! danger
+    The disk space will only be reclaimed after `VACUUM FULL`. You may run out of disk space during the execution of the task or vacuuming if you don't have about 1/3rds of the database size free.
+
+```sh tab="OTP"
+./bin/pleroma_ctl database prune_objects [<options>]
+```
 
-```sh
-$PREFIX pleroma.database prune_objects [<options>]
+```sh tab="From Source"
+mix pleroma.database prune_objects [<options>]
 ```
 
 ### Options
@@ -34,18 +42,30 @@ $PREFIX pleroma.database prune_objects [<options>]
 
 Can be safely re-run
 
-```sh
-$PREFIX bump_all_conversations
+```sh tab="OTP"
+./bin/pleroma_ctl database bump_all_conversations
+```
+
+```sh tab="From Source"
+mix pleroma.database bump_all_conversations
 ```
 
 ## Remove duplicated items from following and update followers count for all users
 
-```sh
-$PREFIX update_users_following_followers_counts
+```sh tab="OTP"
+./bin/pleroma_ctl database update_users_following_followers_counts
+```
+
+```sh tab="From Source"
+mix pleroma.database update_users_following_followers_counts
 ```
 
 ## Fix the pre-existing "likes" collections for all objects
 
-```sh
-$PREFIX fix_likes_collections
+```sh tab="OTP"
+./bin/pleroma_ctl database fix_likes_collections
+```
+
+```sh tab="From Source"
+mix pleroma.database fix_likes_collections
 ```

docs/administration/CLI_tasks/digest.md

 # Managing digest emails
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl digest` and in case of source installs it's `mix pleroma.digest`.
+
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Send digest email since given date (user registration date by default) ignoring user activity status.
 
-```sh
-$PREFIX test <nickname> [<since_date>]
+```sh tab="OTP"
+ ./bin/pleroma_ctl digest test <nickname> [<since_date>]
+```
+
+```sh tab="From Source"
+mix pleroma.digest test <nickname> [<since_date>]
 ```
 
+
 Example: 
-```sh
-$PREFIX test donaldtheduck 2019-05-20
+```sh tab="OTP"
+ ./bin/pleroma_ctl digest test donaldtheduck 2019-05-20
 ```
+
+```sh tab="From Source"
+mix pleroma.digest test donaldtheduck 2019-05-20
+```
+

docs/administration/CLI_tasks/emoji.md

 # Managing emoji packs
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl emoji` and in case of source installs it's `mix pleroma.emoji`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Lists emoji packs and metadata specified in the manifest
 
-```sh
-$PREFIX ls-packs [<options>]
+```sh tab="OTP"
+./bin/pleroma_ctl emoji ls-packs [<options>]
 ```
 
+```sh tab="From Source"
+mix pleroma.emoji ls-packs [<options>]
+```
+
+
 ### Options
 - `-m, --manifest PATH/URL` - path to a custom manifest, it can either be an URL starting with `http`, in that case the manifest will be fetched from that address, or a local path
 
 ## Fetch, verify and install the specified packs from the manifest into `STATIC-DIR/emoji/PACK-NAME`
-```sh
-$PREFIX get-packs [<options>] <packs>
+
+```sh tab="OTP"
+./bin/pleroma_ctl emoji get-packs [<options>] <packs>
+```
+
+```sh tab="From Source"
+mix pleroma.emoji get-packs [<options>] <packs>
 ```
 
 ### Options
 - `-m, --manifest PATH/URL` - same as [`ls-packs`](#ls-packs)
 
 ## Create a new manifest entry and a file list from the specified remote pack file
-```sh
-$PREFIX gen-pack PACK-URL
+
+```sh tab="OTP"
+./bin/pleroma_ctl emoji gen-pack PACK-URL
 ```
+
+```sh tab="From Source"
+mix pleroma.emoji gen-pack PACK-URL
+```
+
 Currently, only .zip archives are recognized as remote pack files and packs are therefore assumed to be zip archives. This command is intended to run interactively and will first ask you some basic questions about the pack, then download the remote file and generate an SHA256 checksum for it, then generate an emoji file list for you. 
 
   The manifest entry will either be written to a newly created `index.json` file or appended to the existing one, *replacing* the old pack with the same name if it was in the file previously.

docs/administration/CLI_tasks/general_cli_task_info.include

+Every command should be ran as the `pleroma` user from it's home directory. For example if you are superuser, you would have to wrap the command in `su pleroma -s $SHELL -lc "$COMMAND"`.
+
+??? note "From source note about `MIX_ENV`"
+
+     The `mix` command should be prefixed with the name of environment your Pleroma server is running in, usually it's `MIX_ENV=prod`

docs/administration/CLI_tasks/instance.md

 # Managing instance configuration
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl instance` and in case of source installs it's `mix pleroma.instance`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Generate a new configuration file
-```sh
-$PREFIX gen [<options>]
+```sh tab="OTP"
+ ./bin/pleroma_ctl instance gen [<options>]
 ```
 
+```sh tab="From Source"
+mix pleroma.instance gen [<options>]
+```
+
+
 If any of the options are left unspecified, you will be prompted interactively.
 
 ### Options

docs/administration/CLI_tasks/relay.md

 # Managing relays
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl relay` and in case of source installs it's `mix pleroma.relay`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Follow a relay
-```sh
-$PREFIX follow <relay_url>
+
+```sh tab="OTP"
+./bin/pleroma_ctl relay follow <relay_url>
 ```
 
-Example:
-```sh
-$PREFIX follow https://example.org/relay
+```sh tab="From Source"
+mix pleroma.relay follow <relay_url>
 ```
 
 ## Unfollow a remote relay
 
-```sh
-$PREFIX unfollow <relay_url>
+```sh tab="OTP"
+./bin/pleroma_ctl relay unfollow <relay_url>
 ```
 
-Example:
-```sh
-$PREFIX unfollow https://example.org/relay
+```sh tab="From Source"
+mix pleroma.relay unfollow <relay_url>
 ```
 
 ## List relay subscriptions
 
-```sh
-$PREFIX list
+```sh tab="OTP"
+./bin/pleroma_ctl relay list
+```
+
+```sh tab="From Source"
+mix pleroma.relay list
 ```

docs/administration/CLI_tasks/uploads.md

 # Managing uploads
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl uploads` and in case of source installs it's `mix pleroma.uploads`.
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
 
 ## Migrate uploads from local to remote storage
-```sh
-$PREFIX migrate_local <target_uploader> [<options>]
+```sh tab="OTP"
+ ./bin/pleroma_ctl uploads migrate_local <target_uploader> [<options>]
 ```
+
+```sh tab="From Source"
+mix pleroma.uploads migrate_local <target_uploader> [<options>]
+```
+
 ### Options
 - `--delete` - delete local uploads after migrating them to the target uploader
 

docs/administration/CLI_tasks/user.md

 # Managing users
 
-Every command should be ran with a prefix, in case of OTP releases it is `./bin/pleroma_ctl user` and in case of source installs it's `mix pleroma.user`.