...
 
Commits (171)
......@@ -8,19 +8,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- OStatus: eliminate the possibility of a protocol downgrade attack.
- OStatus: prevent following locked accounts, bypassing the approval process.
### Removed
- **Breaking:** GNU Social API with Qvitter extensions support
- **Breaking:** ActivityPub: The `accept_blocks` configuration setting.
- Emoji: Remove longfox emojis.
- Remove `Reply-To` header from report emails for admins.
### Changed
- **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
- **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
- **Breaking:** `/api/pleroma/notifications/read` is moved to `/api/v1/pleroma/notifications/read` and now supports `max_id` and responds with Mastodon API entities.
- Configuration: OpenGraph and TwitterCard providers enabled by default
- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
- Mastodon API: `pleroma.thread_muted` key in the Status entity
- Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
- NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
- NodeInfo: Return `mailerEnabled` in `metadata`
- Mastodon API: Unsubscribe followers when they unfollow a user
- AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
- Improve digest email template
– Pagination: (optional) return `total` alongside with `items` when paginating
### Fixed
- Following from Osada
- Not being able to pin unlisted posts
- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
- Favorites timeline doing database-intensive queries
......@@ -28,7 +38,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- `federation_incoming_replies_max_depth` option being ignored in certain cases
- Federation/MediaProxy not working with instances that have wrong certificate order
- Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`)
- Mastodon API: Misskey's endless polls being unable to render
- Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
- Mastodon API: Notifications endpoint crashing if one notification failed to render
- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
- Mastodon API: `muted` in the Status entity, using author's account to determine if the tread was muted
- Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
......@@ -48,6 +60,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
- MRF: fix use of unserializable keyword lists in describe() implementations
- ActivityPub: Deactivated user deletion
- ActivityPub: Fix `/users/:nickname/inbox` crashing without an authenticated user
- MRF: fix ability to follow a relay when AntiFollowbotPolicy was enabled
### Added
- Expiring/ephemeral activites. All activities can have expires_at value set, which controls when they should be deleted automatically.
......@@ -95,6 +109,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mix Tasks: `mix pleroma.database fix_likes_collections`
- Federation: Remove `likes` from objects.
- Admin API: Added moderation log
- Web response cache (currently, enabled for ActivityPub)
- Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`)
### Changed
- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
......@@ -102,10 +118,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- RichMedia: parsers and their order are configured in `rich_media` config.
- RichMedia: add the rich media ttl based on image expiration time.
### Removed
- Emoji: Remove longfox emojis.
- Remove `Reply-To` header from report emails for admins.
- ActivityPub: The `accept_blocks` configuration setting.
## [1.0.1] - 2019-07-14
### Security
......
FROM rinpatch/elixir:1.9.0-rc.0-alpine as build
FROM elixir:1.9-alpine as build
COPY . .
......@@ -12,7 +12,7 @@ RUN apk add git gcc g++ musl-dev make &&\
mkdir release &&\
mix release --path release
FROM alpine:latest
FROM alpine:3.9
ARG HOME=/opt/pleroma
ARG DATA=/var/lib/pleroma
......
......@@ -8,7 +8,7 @@ Pleroma is a microblogging server software that can federate (= exchange message
Pleroma is written in Elixir, high-performance and can run on small devices like a Raspberry Pi.
For clients it supports both the [GNU Social API with Qvitter extensions](https://twitter-api.readthedocs.io/en/latest/index.html) and the [Mastodon client API](https://docs.joinmastodon.org/api/guidelines/).
For clients it supports the [Mastodon client API](https://docs.joinmastodon.org/api/guidelines/) with Pleroma extensions (see "Pleroma's APIs and Mastodon API extensions" section on <https://docs-develop.pleroma.social>).
- [Client Applications for Pleroma](https://docs-develop.pleroma.social/clients.html)
......
......@@ -56,20 +56,6 @@ config :pleroma, Pleroma.Captcha,
seconds_valid: 60,
method: Pleroma.Captcha.Kocaptcha
config :pleroma, :hackney_pools,
federation: [
max_connections: 50,
timeout: 150_000
],
media: [
max_connections: 50,
timeout: 150_000
],
upload: [
max_connections: 25,
timeout: 300_000
]
config :pleroma, Pleroma.Captcha.Kocaptcha, endpoint: "https://captcha.kotobank.ch"
# Upload configuration
......@@ -186,20 +172,13 @@ config :mime, :types, %{
"application/ld+json" => ["activity+json"]
}
config :tesla, adapter: Tesla.Adapter.Hackney
config :tesla, adapter: Tesla.Adapter.Gun
# Configures http settings, upstream proxy etc.
config :pleroma, :http,
proxy_url: nil,
send_user_agent: true,
adapter: [
ssl_options: [
# Workaround for remote server certificate chain issues
partial_chain: &:hackney_connect.partial_chain/1,
# We don't support TLS v1.3 yet
versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"]
]
]
adapter: []
config :pleroma, :instance,
name: "Pleroma",
......@@ -373,6 +352,8 @@ config :pleroma, :chat, enabled: true
config :phoenix, :format_encoders, json: Jason
config :phoenix, :json_library, Jason
config :pleroma, :gopher,
enabled: false,
ip: {0, 0, 0, 0},
......@@ -560,6 +541,24 @@ config :pleroma, :rate_limit, nil
config :pleroma, Pleroma.ActivityExpiration, enabled: true
config :pleroma, :web_cache_ttl,
activity_pub: nil,
activity_pub_question: 30_000
config :pleroma, :gun_pools,
federation: [
max_connections: 50,
timeout: 150_000
],
media: [
max_connections: 50,
timeout: 150_000
],
upload: [
max_connections: 25,
timeout: 300_000
]
# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
import_config "#{Mix.env()}.exs"
......@@ -10,7 +10,7 @@ config :pleroma, :instance,
notify_email: System.get_env("NOTIFY_EMAIL"),
limit: 5000,
registrations_open: false,
dynamic_configuration: true
healthcheck: true
config :pleroma, Pleroma.Repo,
adapter: Ecto.Adapters.Postgres,
......
......@@ -86,6 +86,9 @@ config :joken, default_signer: "yU8uHKq+yyAkZ11Hx//jcdacWc8yQ1bxAAGrplzB0Zwwjkp3
config :pleroma, Pleroma.ReverseProxy.Client, Pleroma.ReverseProxy.ClientMock
config :pleroma, Pleroma.Gun.API, Pleroma.Gun.API.Mock
config :pleroma, Pleroma.Mint.API, Pleroma.Mint.API.Mock
if File.exists?("./config/test.secret.exs") do
import_config "test.secret.exs"
else
......
......@@ -26,6 +26,7 @@ Has these additional fields under the `pleroma` object:
- `content`: a map consisting of alternate representations of the `content` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
- `spoiler_text`: a map consisting of alternate representations of the `spoiler_text` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
- `expires_at`: a datetime (iso8601) that states when the post will expire (be deleted automatically), or empty if the post won't expire
- `thread_muted`: true if the thread the post belongs to is muted
## Attachments
......@@ -90,6 +91,20 @@ Additional parameters can be added to the JSON body/Form data:
- `expires_in`: The number of seconds the posted activity should expire in. When a posted activity expires it will be deleted from the server, and a delete request for it will be federated. This needs to be longer than an hour.
- `in_reply_to_conversation_id`: Will reply to a given conversation, addressing only the people who are part of the recipient set of that conversation. Sets the visibility to `direct`.
## GET `/api/v1/statuses`
An endpoint to get multiple statuses by IDs.
Required parameters:
- `ids`: array of activity ids
Usage example: `GET /api/v1/statuses/?ids[]=1&ids[]=2`.
Returns: array of Status.
The maximum number of statuses is limited to 100 per request.
## PATCH `/api/v1/update_credentials`
Additional parameters can be added to the JSON body/Form data:
......
......@@ -126,13 +126,14 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
## `/api/pleroma/admin/`…
See [Admin-API](Admin-API.md)
## `/api/pleroma/notifications/read`
### Mark a single notification as read
## `/api/v1/pleroma/notifications/read`
### Mark notifications as read
* Method `POST`
* Authentication: required
* Params:
* `id`: notification's id
* Response: JSON. Returns `{"status": "success"}` if the reading was successful, otherwise returns `{"error": "error_msg"}`
* Params (mutually exclusive):
* `id`: a single notification id to read
* `max_id`: read all notifications up to this id
* Response: Notification entity/Array of Notification entities that were read. In case of `max_id`, only the first 80 read notifications will be returned.
## `/api/v1/pleroma/accounts/:id/subscribe`
### Subscribe to receive notifications for all statuses posted by a user
......
......@@ -690,3 +690,12 @@ Supported rate limiters:
* `:relation_id_action` for actions on relation with a specific user (follow, unfollow)
* `:statuses_actions` for create / delete / fav / unfav / reblog / unreblog actions on any statuses
* `:status_id_action` for fav / unfav or reblog / unreblog actions on the same status by the same user
## :web_cache_ttl
The expiration time for the web responses cache. Values should be in milliseconds or `nil` to disable expiration.
Available caches:
* `:activity_pub` - activity pub routes (except question activities). Defaults to `nil` (no expiration).
* `:activity_pub_question` - activity pub routes (question activities). Defaults to `30_000` (30 seconds).
......@@ -71,26 +71,26 @@ server {
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
# this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
# and `localhost.` resolves to [::0] on some systems: see issue #930
# this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
# and `localhost.` resolves to [::0] on some systems: see issue #930
proxy_pass http://127.0.0.1:4000;
client_max_body_size 16m;
}
location ~ ^/(media|proxy) {
proxy_cache pleroma_media_cache;
proxy_cache pleroma_media_cache;
slice 1m;
proxy_cache_key $host$uri$is_args$args$slice_range;
proxy_set_header Range $slice_range;
proxy_http_version 1.1;
proxy_cache_valid 200 206 301 304 1h;
proxy_cache_lock on;
proxy_cache_lock on;
proxy_ignore_client_abort on;
proxy_buffering on;
proxy_buffering on;
chunked_transfer_encoding on;
proxy_ignore_headers Cache-Control;
proxy_hide_header Cache-Control;
proxy_pass http://localhost:4000;
proxy_hide_header Cache-Control;
proxy_pass http://127.0.0.1:4000;
}
}
......@@ -27,7 +27,7 @@ defmodule Mix.Tasks.Pleroma.Benchmark do
})
end
def run(["render_timeline", nickname]) do
def run(["render_timeline", nickname | _] = args) do
start_pleroma()
user = Pleroma.User.get_by_nickname(nickname)
......@@ -37,33 +37,37 @@ defmodule Mix.Tasks.Pleroma.Benchmark do
|> Map.put("blocking_user", user)
|> Map.put("muting_user", user)
|> Map.put("user", user)
|> Map.put("limit", 80)
|> Map.put("limit", 4096)
|> Pleroma.Web.ActivityPub.ActivityPub.fetch_public_activities()
|> Enum.reverse()
inputs = %{
"One activity" => Enum.take_random(activities, 1),
"Ten activities" => Enum.take_random(activities, 10),
"Twenty activities" => Enum.take_random(activities, 20),
"Forty activities" => Enum.take_random(activities, 40),
"Eighty activities" => Enum.take_random(activities, 80)
"1 activity" => Enum.take_random(activities, 1),
"10 activities" => Enum.take_random(activities, 10),
"20 activities" => Enum.take_random(activities, 20),
"40 activities" => Enum.take_random(activities, 40),
"80 activities" => Enum.take_random(activities, 80)
}
inputs =
if Enum.at(args, 2) == "extended" do
Map.merge(inputs, %{
"200 activities" => Enum.take_random(activities, 200),
"500 activities" => Enum.take_random(activities, 500),
"2000 activities" => Enum.take_random(activities, 2000),
"4096 activities" => Enum.take_random(activities, 4096)
})
else
inputs
end
Benchee.run(
%{
"Parallel rendering" => fn activities ->
Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
activities: activities,
for: user,
as: :activity
})
end,
"Standart rendering" => fn activities ->
Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
activities: activities,
for: user,
as: :activity,
parallel: false
as: :activity
})
end
},
......
......@@ -4,6 +4,7 @@
defmodule Mix.Tasks.Pleroma.Emoji do
use Mix.Task
import Mix.Pleroma
@shortdoc "Manages emoji packs"
@moduledoc """
......@@ -56,7 +57,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
"""
def run(["ls-packs" | args]) do
Application.ensure_all_started(:hackney)
start_pleroma()
{options, [], []} = parse_global_opts(args)
......@@ -82,7 +83,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
end
def run(["get-packs" | args]) do
Application.ensure_all_started(:hackney)
start_pleroma()
{options, pack_names, []} = parse_global_opts(args)
......@@ -178,7 +179,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
end
def run(["gen-pack", src]) do
Application.ensure_all_started(:hackney)
start_pleroma()
proposed_name = Path.basename(src) |> Path.rootname()
name = String.trim(IO.gets("Pack name [#{proposed_name}]: "))
......
......@@ -173,6 +173,13 @@ defmodule Pleroma.Activity do
|> Repo.one()
end
def all_by_ids_with_object(ids) do
Activity
|> where([a], a.id in ^ids)
|> with_preloaded_object()
|> Repo.all()
end
def by_object_ap_id(ap_id) do
from(
activity in Activity,
......@@ -308,10 +315,19 @@ defmodule Pleroma.Activity do
%{data: %{"type" => "Create", "object" => %{"id" => ap_id}}} -> ap_id == id
_ -> nil
end)
|> purge_web_resp_cache()
end
def delete_by_ap_id(_), do: nil
defp purge_web_resp_cache(%Activity{} = activity) do
%{path: path} = URI.parse(activity.data["id"])
Cachex.del(:web_resp_cache, path)
activity
end
defp purge_web_resp_cache(nil), do: nil
for {ap_type, type} <- @mastodon_notification_types do
def mastodon_notification_type(%Activity{data: %{"type" => unquote(ap_type)}}),
do: unquote(type)
......@@ -362,12 +378,12 @@ defmodule Pleroma.Activity do
end
def restrict_deactivated_users(query) do
deactivated_users =
from(u in User.Query.build(deactivated: true), select: u.ap_id)
|> Repo.all()
from(activity in query,
where:
fragment(
"? not in (SELECT ap_id FROM users WHERE info->'deactivated' @> 'true')",
activity.actor
)
where: activity.actor not in ^deactivated_users
)
end
......
# Pleroma: A lightweight social networking server
# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Activity.Queries do
@moduledoc """
Contains queries for Activity.
"""
import Ecto.Query, only: [from: 2]
@type query :: Ecto.Queryable.t() | Activity.t()
alias Pleroma.Activity
@spec by_actor(query, String.t()) :: query
def by_actor(query \\ Activity, actor) do
from(
activity in query,
where: fragment("(?)->>'actor' = ?", activity.data, ^actor)
)
end
@spec by_object_id(query, String.t()) :: query
def by_object_id(query \\ Activity, object_id) do
from(activity in query,
where:
fragment(
"coalesce((?)->'object'->>'id', (?)->>'object') = ?",
activity.data,
activity.data,
^object_id
)
)
end
@spec by_type(query, String.t()) :: query
def by_type(query \\ Activity, activity_type) do
from(
activity in query,
where: fragment("(?)->>'type' = ?", activity.data, ^activity_type)
)
end
@spec limit(query, pos_integer()) :: query
def limit(query \\ Activity, limit) do
from(activity in query, limit: ^limit)
end
end
......@@ -39,7 +39,7 @@ defmodule Pleroma.Application do
Pleroma.ActivityExpirationWorker
] ++
cachex_children() ++
hackney_pool_children() ++
gun_pools() ++
[
Pleroma.Web.Federator.RetryQueue,
Pleroma.Stats,
......@@ -95,20 +95,6 @@ defmodule Pleroma.Application do
Pleroma.Web.Endpoint.Instrumenter.setup()
end
def enabled_hackney_pools do
[:media] ++
if Application.get_env(:tesla, :adapter) == Tesla.Adapter.Hackney do
[:federation]
else
[]
end ++
if Pleroma.Config.get([Pleroma.Upload, :proxy_remote]) do
[:upload]
else
[]
end
end
defp cachex_children do
[
build_cachex("used_captcha", ttl_interval: seconds_valid_interval()),
......@@ -116,7 +102,8 @@ defmodule Pleroma.Application do
build_cachex("object", default_ttl: 25_000, ttl_interval: 1000, limit: 2500),
build_cachex("rich_media", default_ttl: :timer.minutes(120), limit: 5000),
build_cachex("scrubber", limit: 2500),
build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500)
build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500),
build_cachex("web_resp", limit: 2500)
]
end
......@@ -157,10 +144,16 @@ defmodule Pleroma.Application do
defp chat_child(_, _), do: []
defp hackney_pool_children do
for pool <- enabled_hackney_pools() do
options = Pleroma.Config.get([:hackney_pools, pool])
:hackney_pool.child_spec(pool, options)
defp gun_pools do
if Application.get_env(:tesla, :adapter) == Tesla.Adapter.Gun || Mix.env() == :test do
for {pool_name, opts} <- Pleroma.Config.get([:gun_pools]) do
%{
id: :"gun_pool_#{pool_name}",
start: {Pleroma.Gun.Connections, :start_link, [{pool_name, opts}]}
}
end
else
[]
end
end
......
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Gun.API do
@callback open(charlist(), pos_integer(), map()) :: {:ok, pid()}
@callback info(pid()) :: map()
@callback close(pid()) :: :ok
@callback await_up(pid) :: {:ok, atom()} | {:error, atom()}
@callback connect(pid(), map()) :: reference()
@callback await(pid(), reference()) :: {:response, :fin, 200, []}
def open(host, port, opts), do: api().open(host, port, opts)
def info(pid), do: api().info(pid)
def close(pid), do: api().close(pid)
def await_up(pid), do: api().await_up(pid)
def connect(pid, opts), do: api().connect(pid, opts)
def await(pid, ref), do: api().await(pid, ref)
defp api, do: Pleroma.Config.get([Pleroma.Gun.API], Pleroma.Gun.API.Gun)
end
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Gun.API.Gun do
@behaviour Pleroma.Gun.API
alias Pleroma.Gun.API
@gun_keys [
:connect_timeout,
:http_opts,
:http2_opts,
:protocols,
:retry,
:retry_timeout,
:trace,
:transport,
:tls_opts,
:tcp_opts,
:ws_opts
]
@impl API
def open(host, port, opts) do
:gun.open(host, port, Map.take(opts, @gun_keys))
end
@impl API
def info(pid), do: :gun.info(pid)
@impl API
def close(pid), do: :gun.close(pid)
@impl API
def await_up(pid), do: :gun.await_up(pid)
@impl API
def connect(pid, opts), do: :gun.connect(pid, opts)
@impl API
def await(pid, ref), do: :gun.await(pid, ref)
end
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Gun.API.Mock do
@behaviour Pleroma.Gun.API
alias Pleroma.Gun.API
@impl API
def open(domain, 80, %{genserver_pid: genserver_pid})
when domain in ['another-domain.com', 'some-domain.com'] do
{:ok, conn_pid} = Task.start_link(fn -> Process.sleep(1_000) end)
Registry.register(API.Mock, conn_pid, %{
origin_scheme: "http",
origin_host: domain,
origin_port: 80
})
send(genserver_pid, {:gun_up, conn_pid, :http})
{:ok, conn_pid}
end
@impl API
def open('some-domain.com', 443, %{genserver_pid: genserver_pid}) do
{:ok, conn_pid} = Task.start_link(fn -> Process.sleep(1_000) end)
Registry.register(API.Mock, conn_pid, %{
origin_scheme: "https",
origin_host: 'some-domain.com',
origin_port: 443
})
send(genserver_pid, {:gun_up, conn_pid, :http2})
{:ok, conn_pid}
end
@impl API
def open('gun_down.com', 80, %{genserver_pid: genserver_pid}) do
{:ok, conn_pid} = Task.start_link(fn -> Process.sleep(1_000) end)
Registry.register(API.Mock, conn_pid, %{
origin_scheme: "http",
origin_host: 'gun_down.com',
origin_port: 80
})
send(genserver_pid, {:gun_down, conn_pid, :http, nil, nil, nil})
{:ok, conn_pid}
end
@impl API
def open('gun_down_and_up.com', 80, %{genserver_pid: genserver_pid}) do
{:ok, conn_pid} = Task.start_link(fn -> Process.sleep(1_000) end)
Registry.register(API.Mock, conn_pid, %{
origin_scheme: "http",
origin_host: 'gun_down_and_up.com',
origin_port: 80
})
send(genserver_pid, {:gun_down, conn_pid, :http, nil, nil, nil})
{:ok, _} =
Task.start_link(fn ->
Process.sleep(500)
send(genserver_pid, {:gun_up, conn_pid, :http})
end)
{:ok, conn_pid}
end
@impl API
def open({127, 0, 0, 1}, 8123, _) do
Task.start_link(fn -> Process.sleep(1_000) end)
end
@impl API
def open('localhost', 9050, _) do
Task.start_link(fn -> Process.sleep(1_000) end)
end
@impl API
def await_up(_pid) do
{:ok, :http}
end
@impl API
def connect(pid, %{host: _, port: 80}) do
ref = make_ref()
Registry.register(API.Mock, ref, pid)
ref
end
@impl API
def connect(pid, %{host: _, port: 443, protocols: [:http2], transport: :tls}) do
ref = make_ref()
Registry.register(API.Mock, ref, pid)
ref
end
@impl API
def await(pid, ref) do
[{_, ^pid}] = Registry.lookup(API.Mock, ref)
{:response, :fin, 200, []}
end
@impl API
def info(pid) do
[{_, info}] = Registry.lookup(API.Mock, pid)
info
end
@impl API
def close(_pid), do: :ok
end
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Gun.Conn do
@moduledoc """
Struct for gun connection data
"""
@type gun_state :: :open | :up | :down
@type conn_state :: :init | :active | :idle
@type t :: %__MODULE__{
conn: pid(),
gun_state: gun_state(),
waiting_pids: [pid()],
conn_state: conn_state(),
used_by: [pid()],
last_reference: pos_integer(),
crf: float()
}
defstruct conn: nil,
gun_state: :open,
waiting_pids: [],
conn_state: :init,
used_by: [],
last_reference: :os.system_time(:second),
crf: 1
end
# Pleroma: A lightweight social networking server
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Gun.Connections do
use GenServer
require Logger
@type domain :: String.t()
@type conn :: Pleroma.Gun.Conn.t()
@type t :: %__MODULE__{
conns: %{domain() => conn()},
opts: keyword()
}
defstruct conns: %{}, opts: [], queue: []
alias Pleroma.Gun.API
alias Pleroma.Gun.Conn
@spec start_link({atom(), keyword()}) :: {:ok, pid()}
def start_link({name, opts}) do
GenServer.start_link(__MODULE__, opts, name: name)
end
@impl true
def init(opts), do: {:ok, %__MODULE__{conns: %{}, opts: opts}}
@spec checkin(String.t(), keyword(), atom()) :: pid()
def checkin(url, opts \\ [], name \\ :default) do
opts = Enum.into(opts, %{})
uri = URI.parse(url)
connect_timeout = if opts[:connect_timeout], do: opts[:connect_timeout], else: 5_000
GenServer.call(
name,
{:checkin, %{opts: opts, uri: uri}},
connect_timeout
)
end
@spec alive?(atom()) :: boolean()
def alive?(name \\ :default) do
pid = Process.whereis(name)
if pid, do: Process.alive?(pid), else: false
end
@spec get_state(atom()) :: t()
def get_state(name \\ :default) do
GenServer.call(name, {:state})
end
def checkout(conn, pid, name \\ :default) do
GenServer.cast(name, {:checkout, conn, pid})
end
def process_queue(name \\ :default) do
GenServer.cast(name, {:process_queue})
end
@impl true
def handle_cast({:checkout, conn_pid, pid}, state) do
{key, conn} = find_conn(state.conns, conn_pid)
used_by = List.keydelete(conn.used_by, pid, 0)
conn_state = if used_by == [], do: :idle, else: conn.conn_state
state = put_in(state.conns[key], %{conn | conn_state: conn_state, used_by: used_by})
{:noreply, state}
end
@impl true
def handle_cast({:process_queue}, state) do
case state.queue do
[{from, key, uri, opts} | _queue] ->
try_to_checkin(key, uri, from, state, Map.put(opts, :from_cast, true))
[] ->
{:noreply, state}
end
end
@impl true
def handle_call({:checkin, %{opts: opts, uri: uri}}, from, state) do
key = compose_key(uri)
case state.conns[key] do
%{conn: conn, gun_state: gun_state} = current_conn when gun_state == :up ->
time = current_time()
last_reference = time - current_conn.last_reference
current_crf = crf(last_reference, 100, current_conn.crf)
state =
put_in(state.conns[key], %{
current_conn
| last_reference: time,
crf: current_crf,
conn_state: :active,
used_by: [from | current_conn.used_by]
})
{:reply, conn, state}
%{gun_state: gun_state, waiting_pids: pids} when gun_state in [:open, :down] ->
state = put_in(state.conns[key].waiting_pids, [from | pids])
{:noreply, state}
nil ->
max_connections = state.opts[:max_connections]
if Enum.count(state.conns) < max_connections do
open_conn(key, uri, from, state, opts)
else
try_to_checkin(key, uri, from, state, opts)
end
end
end
@impl true
def handle_call({:state}, _from, state), do: {:reply, state, state}
@impl true
def handle_info({:gun_up, conn_pid, _protocol}, state) do
conn_key = compose_key_gun_info(conn_pid)
{key, conn} = find_conn(state.conns, conn_pid, conn_key)
# Update state of the current connection and set waiting_pids to empty list
time = current_time()
last_reference = time - conn.last_reference
current_crf = crf(last_reference, 100, conn.crf)
state =
put_in(state.conns[key], %{
conn
| gun_state: :up,
waiting_pids: [],
last_reference: time,
crf: current_crf,
conn_state: :active,
used_by: conn.waiting_pids ++ conn.used_by
})
# Send to all waiting processes connection pid
Enum.each(conn.waiting_pids, fn waiting_pid -> GenServer.reply(waiting_pid, conn_pid) end)
{:noreply, state}
end
@impl true
def handle_info({:gun_down, conn_pid, _protocol, _reason, _killed, _unprocessed}, state) do
# we can't get info on this pid, because pid is dead
{key, conn} = find_conn(state.conns, conn_pid)
Enum.each(conn.waiting_pids, fn waiting_pid -> GenServer.reply(waiting_pid, nil) end)
state = put_in(state.conns[key].gun_state, :down)
{:noreply, state}
end
defp compose_key(uri), do: "#{uri.scheme}:#{uri.host}:#{uri.port}"
defp compose_key_gun_info(pid) do
info = API.info(pid)
"#{info.origin_scheme}:#{info.origin_host}:#{info.origin_port}"
end
defp find_conn(conns, conn_pid) do
Enum.find(conns, fn {_key, conn} ->
conn.conn == conn_pid
end)
end
defp find_conn(conns, conn_pid, conn_key) do
Enum.find(conns, fn {key, conn} ->
key == conn_key and conn.conn == conn_pid
end)
end
defp open_conn(key, uri, from, state, %{proxy: {proxy_host, proxy_port}} = opts) do
host = to_charlist(uri.host)
port = uri.port
tls_opts = Map.get(opts, :tls_opts, [])
connect_opts = %{host: host, port: port}
connect_opts =
if uri.scheme == "https" do
Map.put(connect_opts, :protocols, [:http2])
|> Map.put(:transport, :tls)
|> Map.put(:tls_opts, tls_opts)
else
connect_opts
end
with open_opts <- Map.delete(opts, :tls_opts),
{:ok, conn} <- API.open(proxy_host, proxy_port, open_opts),
{:ok, _} <- API.await_up(conn),
stream <- API.connect(conn, connect_opts),
{:response, :fin, 200, _} <- API.await(conn, stream) do
state =
put_in(state.conns[key], %Conn{
conn: conn,
waiting_pids: [],
gun_state: :up,
conn_state: :active,
used_by: [from]
})
if opts[:from_cast] do
GenServer.reply(from, conn)
end
{:reply, conn, state}
else
error ->
Logger.warn(inspect(error))
{:reply, nil, state}
end
end
defp open_conn(key, uri, from, state, opts) do
host = to_charlist(uri.host)
port = uri.port
with {:ok, conn} <- API.open(host, port, opts) do
state =
if opts[:from_cast] do
put_in(state.queue, List.keydelete(state.queue, from, 0))
else
state
end
state =
put_in(state.conns[key], %Conn{
conn: conn,
waiting_pids: [from]
})
{:noreply, state}
else
error ->
Logger.warn(inspect(error))
{:reply, nil, state}
end
end
defp try_to_checkin(key, uri, from, state, opts) do
unused_conns =
state.conns
|> Enum.filter(fn {_k, v} ->
v.conn_state == :idle and v.waiting_pids == [] and v.used_by == []
end)
|> Enum.sort(fn {_x_k, x}, {_y_k, y} ->
x.crf < y.crf and x.last_reference < y.last_reference
end)
case unused_conns do
[{close_key, least_used} | _conns] ->
:ok = API.close(least_used.conn)
state =
put_in(
state.conns,
Map.delete(state.conns, close_key)
)
open_conn(key, uri, from, state, opts)
[] ->
queue =
if List.keymember?(state.queue, from, 0),
do: state.queue,
else: state.queue ++ [{from, key, uri, opts}]
state = put_in(state.queue, queue)
{:noreply, state}
end
end
defp current_time do
:os.system_time(:second)
end
def crf(current, steps, crf) do
1 + :math.pow(0.5, current / steps) * crf
end
end
......@@ -9,6 +9,7 @@ defmodule Pleroma.Healthcheck do
alias Pleroma.Healthcheck
alias Pleroma.Repo
@derive Jason.Encoder
defstruct pool_size: 0,
active: 0,
idle: 0,
......
......@@ -7,14 +7,13 @@ defmodule Pleroma.HTTP.Connection do
Connection for http-requests.
"""
@hackney_options [
@options [
connect_timeout: 10_000,
recv_timeout: 20_000,
follow_redirect: true,
force_redirect: true,
timeout: 20_000,
pool: :federation
]
@adapter Application.get_env(:tesla, :adapter)
require Logger
@doc """
Configure a client connection
......@@ -25,19 +24,114 @@ defmodule Pleroma.HTTP.Connection do
"""
@spec new(Keyword.t()) :: Tesla.Env.client()
def new(opts \\ []) do
Tesla.client([], {@adapter, hackney_options(opts)})
middleware = [Tesla.Middleware.FollowRedirects]
adapter = Application.get_env(:tesla, :adapter)
Tesla.client(middleware, {adapter, options(opts)})
end
# fetch Hackney options
# fetch http options
#
def hackney_options(opts) do
def options(opts) do
options = Keyword.get(opts, :adapter, [])
adapter_options = Pleroma.Config.get([:http, :adapter], [])
proxy_url = Pleroma.Config.get([:http, :proxy_url], nil)
@hackney_options
|> Keyword.merge(adapter_options)
|> Keyword.merge(options)
|> Keyword.merge(proxy: proxy_url)
proxy =
case parse_proxy(proxy_url) do
{:ok, proxy_host, proxy_port} -> {proxy_host, proxy_port}
_ -> nil
end
options =
@options
|> Keyword.merge(adapter_options)
|> Keyword.merge(options)
options = if proxy, do: Keyword.put(options, :proxy, proxy), else: options
pool = options[:pool]
url = options[:url]
if not is_nil(url) and not is_nil(pool) and Pleroma.Gun.Connections.alive?(pool) do
get_conn_for_gun(url, options, pool)
else
options
end
end
defp get_conn_for_gun(url, options, pool) do
try do
case Pleroma.Gun.Connections.checkin(url, options, pool) do
nil ->
options
conn ->
%{host: host, port: port} = URI.parse(url)
Keyword.put(options, :conn, conn)
|> Keyword.put(:close_conn, false)
|> Keyword.put(:original, "#{host}:#{port}")
end
rescue
error ->
Logger.warn("Gun pool checkin caused error #{inspect(error)}")
options
catch
:exit, error ->
Logger.warn("Gun pool checkin exited with error #{inspect(error)}")
options
end
end
@spec parse_proxy(String.t() | tuple() | nil) ::
{tuple, pos_integer()} | {:error, atom()} | nil
def parse_proxy(nil), do: nil
def parse_proxy(proxy) when is_binary(proxy) do
with [host, port] <- String.split(proxy, ":"),
{port, ""} <- Integer.parse(port) do
{:ok, parse_host(host, transform_fun()), port}
else
{_, _} ->
Logger.warn("parsing port in proxy fail #{inspect(proxy)}")
{:error, :error_parsing_port_in_proxy}
:error ->
Logger.warn("parsing port in proxy fail #{inspect(proxy)}")
{:error, :error_parsing_port_in_proxy}
_ ->
Logger.warn("parsing proxy fail #{inspect(proxy)}")
{:error, :error_parsing_proxy}
end
end
def parse_proxy(proxy) when is_tuple(proxy) do
with {_type, host, port} <- proxy do
{:ok, parse_host(host, transform_fun()), port}
else
_ ->
Logger.warn("parsing proxy fail #{inspect(proxy)}")
{:error, :error_parsing_proxy}
end
end
@spec parse_host(String.t() | tuple(), fun()) :: charlist() | atom()
def parse_host(host, transform_fun) when is_atom(host), do: transform_fun.(host)
def parse_host(host, transform_fun) when is_binary(host) do
host = transform_fun.(host)
case :inet.parse_address(host) do
{:error, :einval} -> host
{:ok, ip} -> ip
end
end
defp transform_fun do
adapter = Application.get_env(:tesla, :adapter)
if adapter == Tesla.Adapter.Gun, do: &to_charlist/1, else: &to_string/1
end
end
......@@ -27,22 +27,46 @@ defmodule Pleroma.HTTP do
"""
def request(method, url, body \\ "", headers \\ [], options \\ []) do
try do
options =
process_request_options(options)
|> process_sni_options(url)
url = if url, do: URI.encode(url), else: url
options = process_request_options(options)
adapter_gun? = Application.get_env(:tesla, :adapter) == Tesla.Adapter.Gun
options =
if adapter_gun? do
process_gun_adapter_options(url, options)
else
options
end
params = Keyword.get(options, :params, [])
params = Keyword.get(options, :params, [])
request =
%{}
|> Builder.method(method)
|> Builder.url(url)
|> Builder.headers(headers)
|> Builder.opts(options)
|> Builder.url(url)
|> Builder.add_param(:body, :body, body)
|> Builder.add_param(:query, :query, params)
|> Enum.into([])
|> (&Tesla.request(Connection.new(options), &1)).()
client = Connection.new(options)
try do
response = Tesla.request(client, request)
if adapter_gun? do
%{adapter: {_, _, [adapter_options]}} = client
if conn = adapter_options[:conn] do
pool = adapter_options[:pool]
Pleroma.Gun.Connections.checkout(conn, self(), pool)
Pleroma.Gun.Connections.process_queue(pool)
end
end
response
rescue
e ->
{:error, e}
......@@ -52,20 +76,8 @@ defmodule Pleroma.HTTP do
end
end
defp process_sni_options(options, nil), do: options
defp process_sni_options(options, url) do
uri = URI.parse(url)
host = uri.host |> to_charlist()
case uri.scheme do
"https" -> options ++ [ssl: [server_name_indication: host]]
_ -> options
end
end
def process_request_options(options) do
Keyword.merge(Pleroma.HTTP.Connection.hackney_options([]), options)
Keyword.merge(Pleroma.HTTP.Connection.options([]), options)
end
@doc """
......@@ -83,4 +95,49 @@ defmodule Pleroma.HTTP do
"""
def post(url, body, headers \\ [], options \\ []),
do: request(:post, url, body, headers, options)
defp process_gun_adapter_options(nil, options), do: options
defp process_gun_adapter_options(url, options) do
uri = URI.parse(url)
adapter_opts = Keyword.get(options, :adapter, [])
tls_opts = Keyword.get(adapter_opts, :tls_opts, [])
tls_opts =
if uri.scheme == "https" do
host = to_charlist(uri.host)
# verify certificates opts for gun
verify_opts = [
verify: :verify_peer,
cacerts: :certifi.cacerts(),
depth: 20,
server_name_indication: host,
reuse_sessions: false,
verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: host]}
]
Keyword.merge(tls_opts, verify_opts)
else
tls_opts
end
adapter_opts =
if uri.scheme == "https" and uri.port != 443,
do: Map.put(adapter_opts, :transport, :tls),
else: adapter_opts
adapter_opts =
Keyword.put(adapter_opts, :url, url)
|> Keyword.put(:tls_opts, tls_opts)
adapter_opts =
if uri.scheme == "https" do
Keyword.put(adapter_opts, :certificates_verification, true)
else
adapter_opts
end
Keyword.put(options, :adapter, adapter_opts)
end
end