Merge branch 'release/1.0.5' into 'master'

1.0.5 release

See merge request pleroma/pleroma!1549

CHANGELOG.md

@@ -3,6 +3,32 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [1.0.5] - 2019-08-13
+### Fixed
+- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
+- Mastodon API: `muted` in the Status entity, using author's account to determine if the thread was muted
+- Mastodon API: return the actual profile URL in the Account entity's `url` property when appropriate
+- Templates: properly style anchor tags
+- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
+- Not being able to access the Mastodon FE login page on private instances
+- MRF: ensure that subdomain_match calls are case-insensitive
+- Fix internal server error when using the healthcheck API.
+
+### Added
+- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo.
+  Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules.
+- Relays: Added a task to list relay subscriptions.
+- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`)
+- MRF (Simple Policy): Support for wildcard domains.
+- Support for wildcard domains in user domain blocks setting.
+- Configuration: `quarantined_instances` support wildcard domains.
+- Mix Tasks: `mix pleroma.database fix_likes_collections`
+- Configuration: `federation_incoming_replies_max_depth` option
+
+### Removed
+- Federation: Remove `likes` from objects.
+- ActivityPub: The `accept_blocks` configuration setting.
+
 ## [1.0.4] - 2019-08-01
 ### Fixed
 - Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag

config/config.exs

@@ -220,6 +220,7 @@ config :pleroma, :instance,
   },
   registrations_open: true,
   federating: true,
+  federation_incoming_replies_max_depth: 100,
   federation_reachability_timeout_days: 7,
   federation_publisher_modules: [
     Pleroma.Web.ActivityPub.Publisher,
@@ -300,7 +301,6 @@ config :pleroma, :assets,
   default_mascot: :pleroma_fox_tan
 
 config :pleroma, :activitypub,
-  accept_blocks: true,
   unfollow_blocked: true,
   outgoing_blocks: true,
   follow_handshake_timeout: 500
@@ -334,6 +334,10 @@ config :pleroma, :mrf_keyword,
 
 config :pleroma, :mrf_subchain, match_actor: %{}
 
+config :pleroma, :mrf_vocabulary,
+  accept: [],
+  reject: []
+
 config :pleroma, :rich_media,
   enabled: true,
   ignore_hosts: [],

docs/config.md

@@ -87,6 +87,7 @@ config :pleroma, Pleroma.Emails.Mailer,
 * `invites_enabled`: Enable user invitations for admins (depends on `registrations_open: false`).
 * `account_activation_required`: Require users to confirm their emails before signing in.
 * `federating`: Enable federation with other instances
+* `federation_incoming_replies_max_depth`: Max. depth of reply-to activities fetching on incoming federation, to prevent out-of-memory situations while fetching very long threads. If set to `nil`, threads of any depth will be fetched. Lower this value if you experience out-of-memory crashes.
 * `federation_reachability_timeout_days`: Timeout (in days) of each external federation target being unreachable prior to pausing federating to it.
 * `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance
 * `rewrite_policy`: Message Rewrite Policy, either one or a list. Here are the ones available by default:
@@ -98,6 +99,7 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See ``:mrf_rejectnonpublic`` section)
   * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
   * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
+  * `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (see `:mrf_vocabulary` section)
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``
@@ -266,6 +268,10 @@ config :pleroma, :mrf_subchain,
 * `federated_timeline_removal`: A list of patterns which result in message being removed from federated timelines (a.k.a unlisted), each pattern can be a string or a [regular expression](https://hexdocs.pm/elixir/Regex.html)
 * `replace`: A list of tuples containing `{pattern, replacement}`, `pattern` can be a string or a [regular expression](https://hexdocs.pm/elixir/Regex.html)
 
+## :mrf_vocabulary
+* `accept`: A list of ActivityStreams terms to accept.  If empty, all supported messages are accepted.
+* `reject`: A list of ActivityStreams terms to reject.  If empty, no messages are rejected.
+
 ## :media_proxy
 * `enabled`: Enables proxying of remote media to the instance’s proxy
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
@@ -319,7 +325,6 @@ config :pleroma, Pleroma.Web.Endpoint,
 This will make Pleroma listen on `127.0.0.1` port `8080` and generate urls starting with `https://example.com:2020`
 
 ## :activitypub
-* ``accept_blocks``: Whether to accept incoming block activities from other instances
 * ``unfollow_blocked``: Whether blocks result in people getting unfollowed
 * ``outgoing_blocks``: Whether to federate blocks to other instances
 * ``deny_follow_blocked``: Whether to disallow following an account that has blocked the user in question

lib/mix/tasks/pleroma/database.ex

@@ -35,6 +35,10 @@ defmodule Mix.Tasks.Pleroma.Database do
   ## Remove duplicated items from following and update followers count for all users
 
       mix pleroma.database update_users_following_followers_counts
+
+  ## Fix the pre-existing "likes" collections for all objects
+
+      mix pleroma.database fix_likes_collections
   """
   def run(["remove_embedded_objects" | args]) do
     {options, [], []} =
@@ -119,4 +123,36 @@ defmodule Mix.Tasks.Pleroma.Database do
       )
     end
   end
+
+  def run(["fix_likes_collections"]) do
+    import Ecto.Query
+
+    start_pleroma()
+
+    from(object in Object,
+      where: fragment("(?)->>'likes' is not null", object.data),
+      select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
+    )
+    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Stream.each(fn objects ->
+      ids =
+        objects
+        |> Enum.filter(fn object -> object.likes |> Jason.decode!() |> is_map() end)
+        |> Enum.map(& &1.id)
+
+      Object
+      |> where([object], object.id in ^ids)
+      |> update([object],
+        set: [
+          data:
+            fragment(
+              "jsonb_set(?, '{likes}', '[]'::jsonb, true)",
+              object.data
+            )
+        ]
+      )
+      |> Repo.update_all([], timeout: :infinity)
+    end)
+    |> Stream.run()
+  end
 end

lib/mix/tasks/pleroma/instance.ex

@@ -34,6 +34,8 @@ defmodule Mix.Tasks.Pleroma.Instance do
   - `--db-configurable Y/N` - Allow/disallow configuring instance from admin part
   - `--uploads-dir` - the directory uploads go in when using a local uploader
   - `--static-dir` - the directory custom public files should be read from (custom emojis, frontend bundle overrides, robots.txt, etc.)
+  - `--listen-ip` - the ip the app should listen to, defaults to 127.0.0.1
+  - `--listen-port` - the port the app should listen to, defaults to 4000
   """
 
   def run(["gen" | rest]) do
@@ -56,7 +58,9 @@ defmodule Mix.Tasks.Pleroma.Instance do
           indexable: :string,
           db_configurable: :string,
           uploads_dir: :string,
-          static_dir: :string
+          static_dir: :string,
+          listen_ip: :string,
+          listen_port: :string
         ],
         aliases: [
           o: :output,
@@ -146,10 +150,26 @@ defmodule Mix.Tasks.Pleroma.Instance do
           "n"
         ) === "y"
 
+      listen_port =
+        get_option(
+          options,
+          :listen_port,
+          "What port will the app listen to (leave it if you are using the default setup with nginx)?",
+          4000
+        )
+
+      listen_ip =
+        get_option(
+          options,
+          :listen_ip,
+          "What ip will the app listen to (leave it if you are using the default setup with nginx)?",
+          "127.0.0.1"
+        )
+
       uploads_dir =
         get_option(
           options,
-          :upload_dir,
+          :uploads_dir,
           "What directory should media uploads go in (when using the local uploader)?",
           Pleroma.Config.get([Pleroma.Uploaders.Local, :uploads])
         )
@@ -186,7 +206,9 @@ defmodule Mix.Tasks.Pleroma.Instance do
           db_configurable?: db_configurable?,
           static_dir: static_dir,
           uploads_dir: uploads_dir,
-          rum_enabled: rum_enabled
+          rum_enabled: rum_enabled,
+          listen_ip: listen_ip,
+          listen_port: listen_port
         )
 
       result_psql =

lib/mix/tasks/pleroma/relay.ex

@@ -5,6 +5,7 @@
 defmodule Mix.Tasks.Pleroma.Relay do
   use Mix.Task
   import Mix.Pleroma
+  alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Relay
 
   @shortdoc "Manages remote relays"
@@ -22,6 +23,10 @@ defmodule Mix.Tasks.Pleroma.Relay do
   ``mix pleroma.relay unfollow <relay_url>``
 
   Example: ``mix pleroma.relay unfollow https://example.org/relay``
+
+  ## List relay subscriptions
+
+  ``mix pleroma.relay list``
   """
   def run(["follow", target]) do
     start_pleroma()
@@ -44,4 +49,19 @@ defmodule Mix.Tasks.Pleroma.Relay do
       {:error, e} -> shell_error("Error while following #{target}: #{inspect(e)}")
     end
   end
+
+  def run(["list"]) do
+    start_pleroma()
+
+    with %User{} = user <- Relay.get_actor() do
+      user.following
+      |> Enum.each(fn entry ->
+        URI.parse(entry)
+        |> Map.get(:host)
+        |> shell_info()
+      end)
+    else
+      e -> shell_error("Error while fetching relay subscription list: #{inspect(e)}")
+    end
+  end
 end

lib/mix/tasks/pleroma/user.ex

@@ -31,8 +31,8 @@ defmodule Mix.Tasks.Pleroma.User do
       mix pleroma.user invite [OPTION...]
 
     Options:
-    - `--expires_at DATE` - last day on which token is active (e.g. "2019-04-05")
-    - `--max_use NUMBER` - maximum numbers of token uses
+    - `--expires-at DATE` - last day on which token is active (e.g. "2019-04-05")
+    - `--max-use NUMBER` - maximum numbers of token uses
 
   ## List generated invites
 

lib/pleroma/object.ex

@@ -44,20 +44,20 @@ defmodule Pleroma.Object do
     Repo.one(from(object in Object, where: fragment("(?)->>'id' = ?", object.data, ^ap_id)))
   end
 
-  def normalize(_, fetch_remote \\ true)
+  def normalize(_, fetch_remote \\ true, options \\ [])
   # If we pass an Activity to Object.normalize(), we can try to use the preloaded object.
   # Use this whenever possible, especially when walking graphs in an O(N) loop!
-  def normalize(%Object{} = object, _), do: object
-  def normalize(%Activity{object: %Object{} = object}, _), do: object
+  def normalize(%Object{} = object, _, _), do: object
+  def normalize(%Activity{object: %Object{} = object}, _, _), do: object
 
   # A hack for fake activities
-  def normalize(%Activity{data: %{"object" => %{"fake" => true} = data}}, _) do
+  def normalize(%Activity{data: %{"object" => %{"fake" => true} = data}}, _, _) do
     %Object{id: "pleroma:fake_object_id", data: data}
   end
 
   # Catch and log Object.normalize() calls where the Activity's child object is not
   # preloaded.
-  def normalize(%Activity{data: %{"object" => %{"id" => ap_id}}}, fetch_remote) do
+  def normalize(%Activity{data: %{"object" => %{"id" => ap_id}}}, fetch_remote, _) do
     Logger.debug(
       "Object.normalize() called without preloaded object (#{ap_id}).  Consider preloading the object!"
     )
@@ -67,7 +67,7 @@ defmodule Pleroma.Object do
     normalize(ap_id, fetch_remote)
   end
 
-  def normalize(%Activity{data: %{"object" => ap_id}}, fetch_remote) do
+  def normalize(%Activity{data: %{"object" => ap_id}}, fetch_remote, _) do
     Logger.debug(
       "Object.normalize() called without preloaded object (#{ap_id}).  Consider preloading the object!"
     )
@@ -78,10 +78,14 @@ defmodule Pleroma.Object do
   end
 
   # Old way, try fetching the object through cache.
-  def normalize(%{"id" => ap_id}, fetch_remote), do: normalize(ap_id, fetch_remote)
-  def normalize(ap_id, false) when is_binary(ap_id), do: get_cached_by_ap_id(ap_id)
-  def normalize(ap_id, true) when is_binary(ap_id), do: Fetcher.fetch_object_from_id!(ap_id)
-  def normalize(_, _), do: nil
+  def normalize(%{"id" => ap_id}, fetch_remote, _), do: normalize(ap_id, fetch_remote)
+  def normalize(ap_id, false, _) when is_binary(ap_id), do: get_cached_by_ap_id(ap_id)
+
+  def normalize(ap_id, true, options) when is_binary(ap_id) do
+    Fetcher.fetch_object_from_id!(ap_id, options)
+  end
+
+  def normalize(_, _, _), do: nil
 
   # Owned objects can only be mutated by their owner
   def authorize_mutation(%Object{data: %{"actor" => actor}}, %User{ap_id: ap_id}),

lib/pleroma/object/fetcher.ex

@@ -22,7 +22,7 @@ defmodule Pleroma.Object.Fetcher do
 
   # TODO:
   # This will create a Create activity, which we need internally at the moment.
-  def fetch_object_from_id(id) do
+  def fetch_object_from_id(id, options \\ []) do
     if object = Object.get_cached_by_ap_id(id) do
       {:ok, object}
     else
@@ -39,7 +39,7 @@ defmodule Pleroma.Object.Fetcher do
              "object" => data
            },
            {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
-           {:ok, activity} <- Transmogrifier.handle_incoming(params),
+           {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
            {:object, _data, %Object{} = object} <-
              {:object, data, Object.normalize(activity, false)} do
         {:ok, object}
@@ -69,8 +69,8 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
-  def fetch_object_from_id!(id) do
-    with {:ok, object} <- fetch_object_from_id(id) do
+  def fetch_object_from_id!(id, options \\ []) do
+    with {:ok, object} <- fetch_object_from_id(id, options) do
       object
     else
       _e ->

lib/pleroma/user.ex

@@ -836,15 +836,15 @@ defmodule Pleroma.User do
   def mutes?(nil, _), do: false
   def mutes?(user, %{ap_id: ap_id}), do: Enum.member?(user.info.mutes, ap_id)
 
-  def blocks?(user, %{ap_id: ap_id}) do
-    blocks = user.info.blocks
-    domain_blocks = user.info.domain_blocks
+  def blocks?(%User{info: info} = _user, %{ap_id: ap_id}) do
+    blocks = info.blocks
+
+    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(info.domain_blocks)
+
     %{host: host} = URI.parse(ap_id)
 
     Enum.member?(blocks, ap_id) ||
-      Enum.any?(domain_blocks, fn domain ->
-        host == domain
-      end)
+      Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
   end
 
   def subscribed_to?(user, %{ap_id: ap_id}) do

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -274,6 +274,9 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     else
       {:fake, true, activity} ->
         {:ok, activity}
+
+      {:error, message} ->
+        {:error, message}
     end
   end
 
@@ -730,8 +733,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_favorited_by(query, %{"favorited_by" => ap_id}) do
     from(
-      activity in query,
-      where: fragment(~s(? <@ (? #> '{"object","likes"}'\)), ^ap_id, activity.data)
+      [_activity, object] in query,
+      where: fragment("(?)->'likes' \\? (?)", object.data, ^ap_id)
     )
   end
 

lib/pleroma/web/activity_pub/mrf.ex

@@ -25,4 +25,46 @@ defmodule Pleroma.Web.ActivityPub.MRF do
   defp get_policies(policy) when is_atom(policy), do: [policy]
   defp get_policies(policies) when is_list(policies), do: policies
   defp get_policies(_), do: []
+
+  @spec subdomains_regex([String.t()]) :: [Regex.t()]
+  def subdomains_regex(domains) when is_list(domains) do
+    for domain <- domains, do: ~r(^#{String.replace(domain, "*.", "(.*\\.)*")}$)i
+  end
+
+  @spec subdomain_match?([Regex.t()], String.t()) :: boolean()
+  def subdomain_match?(domains, host) do
+    Enum.any?(domains, fn domain -> Regex.match?(domain, host) end)
+  end
+
+  @callback describe() :: {:ok | :error, Map.t()}
+
+  def describe(policies) do
+    {:ok, policy_configs} =
+      policies
+      |> Enum.reduce({:ok, %{}}, fn
+        policy, {:ok, data} ->
+          {:ok, policy_data} = policy.describe()
+          {:ok, Map.merge(data, policy_data)}
+
+        _, error ->
+          error
+      end)
+
+    mrf_policies =
+      get_policies()
+      |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
+
+    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+
+    base =
+      %{
+        mrf_policies: mrf_policies,
+        exclusions: length(exclusions) > 0
+      }
+      |> Map.merge(policy_configs)
+
+    {:ok, base}
+  end
+
+  def describe, do: get_policies() |> describe()
 end

lib/pleroma/web/activity_pub/mrf/anti_followbot_policy.ex

@@ -62,4 +62,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiFollowbotPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/anti_link_spam_policy.ex

@@ -5,6 +5,8 @@
 defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
   alias Pleroma.User
 
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
   require Logger
 
   # has the user successfully posted before?
@@ -22,6 +24,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
 
   defp contains_links?(_), do: false
 
+  @impl true
   def filter(%{"type" => "Create", "actor" => actor, "object" => object} = message) do
     with {:ok, %User{} = u} <- User.get_or_fetch_by_ap_id(actor),
          {:contains_links, true} <- {:contains_links, contains_links?(object)},
@@ -45,4 +48,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
 
   # in all other cases, pass through
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/drop_policy.ex

@@ -12,4 +12,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.DropPolicy do
     Logger.info("REJECTING #{inspect(object)}")
     {:reject, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/ensure_re_prepended.ex

@@ -42,4 +42,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.EnsureRePrepended do
   end
 
   def filter(object), do: {:ok, object}
+
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/hellthread_policy.ex

@@ -87,4 +87,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{mrf_hellthread: Pleroma.Config.get([:mrf_hellthread])}}
 end

lib/pleroma/web/activity_pub/mrf/keyword_policy.ex

@@ -94,4 +94,36 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe do
+    # This horror is needed to convert regex sigils to strings
+    mrf_keyword =
+      Pleroma.Config.get(:mrf_keyword, [])
+      |> Enum.map(fn {key, value} ->
+        {key,
+         Enum.map(value, fn
+           {pattern, replacement} ->
+             %{
+               "pattern" =>
+                 if not is_binary(pattern) do
+                   inspect(pattern)
+                 else
+                   pattern
+                 end,
+               "replacement" => replacement
+             }
+
+           pattern ->
+             if not is_binary(pattern) do
+               inspect(pattern)
+             else
+               pattern
+             end
+         end)}
+      end)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_keyword: mrf_keyword}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/no_placeholder_text_policy.ex

@@ -27,4 +27,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.NoPlaceholderTextPolicy do
 
   @impl true
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/noop_policy.ex

@@ -10,4 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.NoOpPolicy do
   def filter(object) do
     {:ok, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end