Commit 008499f6 authored by feld's avatar feld
Browse files

Merge branch 'develop' into 'fix/2412-filters'

# Conflicts:
#   CHANGELOG.md
parents 39335d42 c3dd860a
Pipeline #34648 passed with stages
in 12 minutes and 56 seconds
......@@ -34,6 +34,14 @@ build:
- mix deps.get
- mix compile --force
spec-build:
stage: test
artifacts:
paths:
- spec.json
script:
- mix pleroma.openapi_spec spec.json
benchmark:
stage: benchmark
when: manual
......@@ -155,6 +163,20 @@ review_app:
- (ssh -t dokku@pleroma.online -- certs:add "$CI_ENVIRONMENT_SLUG" /home/dokku/server.crt /home/dokku/server.key) || true
- git push -f dokku@pleroma.online:$CI_ENVIRONMENT_SLUG $CI_COMMIT_SHA:refs/heads/master
spec-deploy:
stage: deploy
artifacts:
paths:
- spec.json
only:
- develop@pleroma/pleroma
image: alpine:latest
before_script:
- apk add curl
script:
- curl -X POST -F"token=$API_DOCS_PIPELINE_TRIGGER" -F'ref=master' -F"variables[BRANCH]=$CI_COMMIT_REF_NAME" -F"variables[JOB_REF]=CI_JOB_ID" https://git.pleroma.social/api/v4/projects/1130/trigger/pipeline
stop_review_app:
image: alpine:3.9
stage: deploy
......
......@@ -10,17 +10,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- **Breaking**: Changed `mix pleroma.user toggle_confirmed` to `mix pleroma.user confirm`
- **Breaking**: Changed `mix pleroma.user toggle_activated` to `mix pleroma.user activate/deactivate`
- **Breaking**: AdminAPI changed User field `confirmation_pending` to `is_confirmed`
- **Breaking**: AdminAPI changed User field `approval_pending` to `is_approved`
- **Breaking**: AdminAPI changed User field `deactivated` to `is_active`
- Polls now always return a `voters_count`, even if they are single-choice.
- Admin Emails: The ap id is used as the user link in emails now.
- Improved registration workflow for email confirmation and account approval modes.
- Search: When using Postgres 11+, Pleroma will use the `websearch_to_tsvector` function to parse search queries.
- Emoji: Support the full Unicode 13.1 set of Emoji for reactions, plus regional indicators.
- Admin API: Reports now ordered by newest
- Deprecated `Pleroma.Uploaders.S3, :public_endpoint`. Now `Pleroma.Upload, :base_url` is the standard configuration key for all uploaders.
- Improved Apache webserver support: updated sample configuration, MediaProxy cache invalidation verified with the included sample script
- Improve OAuth 2.0 provider support. A missing `fqn` field was added to the response, but does not expose the user's email address.
<details>
<summary>API Changes</summary>
- **Breaking:** AdminAPI changed User field `confirmation_pending` to `is_confirmed`
- **Breaking:** AdminAPI changed User field `approval_pending` to `is_approved`
- **Breaking**: AdminAPI changed User field `deactivated` to `is_active`
- **Breaking:** AdminAPI `GET /api/pleroma/admin/users/:nickname_or_id/statuses` changed response format and added the number of total users posts.
- **Breaking:** AdminAPI `GET /api/pleroma/admin/instances/:instance/statuses` changed response format and added the number of total users posts.
- Admin API: Reports now ordered by newest
</details>
### Added
......@@ -38,7 +47,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- OAuth improvements and fixes: more secure session-based authentication (by token that could be revoked anytime), ability to revoke belonging OAuth token from any client etc.
- Ability to set ActivityPub aliases for follower migration.
- Configurable background job limits for RichMedia (link previews) and MediaProxyWarmingPolicy
- Ability to define custom HTTP headers per each frontend
<details>
<summary>API Changes</summary>
......@@ -48,7 +57,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mastodon API: User and conversation mutes can now auto-expire if `expires_in` parameter was given while adding the mute.
- Admin API: An endpoint to manage frontends.
- Streaming API: Add follow relationships updates.
- WebPush: Introduce `pleroma:chat_mention` and `pleroma:emoji_reaction` notification types
- WebPush: Introduce `pleroma:chat_mention` and `pleroma:emoji_reaction` notification types.
- Mastodon API: Add monthly active users to `/api/v1/instance` (`pleroma.stats.mau`).
- Mastodon API: Home, public, hashtag & list timelines accept `only_media`, `remote` & `local` parameters for filtration.
- Mastodon API: `/api/v1/accounts/:id` & `/api/v1/mutes` endpoints accept `with_relationships` parameter and return filled `pleroma.relationship` field.
</details>
### Fixed
......@@ -58,12 +70,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Creating incorrect IPv4 address-style HTTP links when encountering certain numbers.
- Reblog API Endpoint: Do not set visibility parameter to public by default and let CommonAPI to infer it from status, so a user can reblog their private status without explicitly setting reblog visibility to private.
- Tag URLs in statuses are now absolute
- Removed duplicate jobs to purge expired activities
- File extensions of some attachments were incorrectly changed. This feature has been disabled for now.
- Mix task pleroma.instance creates missing parent directories if the configuration or SQL output paths are changed.
<details>
<summary>API Changes</summary>
- Mastodon API: Current user is now included in conversation if it's the only participant.
- Mastodon API: Fixed last_status.account being not filled with account data.
- Mastodon API: Fix not being able to add or remove multiple users at once in lists.
- Mastodon API: Fixed own_votes being not returned with poll data.
- Mastodon API: Fixed creation of scheduled posts with polls.
- Mastodon API: Support for expires_in/expires_at in the Filters.
</details>
......
......@@ -726,7 +726,10 @@
"git" => "https://git.pleroma.social/pleroma/fedi-fe",
"build_url" =>
"https://git.pleroma.social/pleroma/fedi-fe/-/jobs/artifacts/${ref}/download?job=build",
"ref" => "master"
"ref" => "master",
"custom-http-headers" => [
{"service-worker-allowed", "/"}
]
},
"admin-fe" => %{
"name" => "admin-fe",
......
......@@ -60,6 +60,12 @@
label: "Build directory",
type: :string,
description: "The directory inside the zip file "
},
%{
key: "custom-http-headers",
label: "Custom HTTP headers",
type: {:list, :string},
description: "The custom HTTP headers for the frontend"
}
]
......@@ -3218,6 +3224,12 @@
type: :string,
description: "S3 host",
suggestions: ["s3.eu-central-1.amazonaws.com"]
},
%{
key: :region,
type: :string,
description: "S3 region (for AWS)",
suggestions: ["us-east-1"]
}
]
},
......
See `Authentication` section of [the configuration cheatsheet](../configuration/cheatsheet.md#authentication).
......@@ -893,6 +893,22 @@ Pleroma account will be created with the same name as the LDAP user name.
Note, if your LDAP server is an Active Directory server the correct value is commonly `uid: "cn"`, but if you use an
OpenLDAP server the value may be `uid: "uid"`.
### :oauth2 (Pleroma as OAuth 2.0 provider settings)
OAuth 2.0 provider settings:
* `token_expires_in` - The lifetime in seconds of the access token.
* `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
OAuth 2.0 provider and related endpoints:
* `POST /api/v1/apps` creates client app basing on provided params.
* `GET/POST /oauth/authorize` renders/submits authorization form.
* `POST /oauth/token` creates/renews OAuth token.
* `POST /oauth/revoke` revokes provided OAuth token.
* `GET /api/v1/accounts/verify_credentials` (with proper `Authorization` header or `access_token` URI param) returns user info on requester (with `acct` field containing local nickname and `fqn` field containing fully-qualified nickname which could generally be used as email stub for OAuth software that demands email field in identity endpoint response, like Peertube).
### OAuth consumer mode
OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.).
......@@ -965,14 +981,6 @@ config :ueberauth, Ueberauth,
]
```
### OAuth 2.0 provider - :oauth2
Configure OAuth 2 provider capabilities:
* `token_expires_in` - The lifetime in seconds of the access token.
* `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
## Link parsing
### :uri_schemes
......
......@@ -287,7 +287,18 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- *optional* `with_reblogs`: `true`/`false` – allows to see reblogs (default is false)
- Response:
- On failure: `Not found`
- On success: JSON array of user's latest statuses
- On success: JSON, where:
- `total`: total count of the statuses for the user
- `activities`: list of the statuses for the user
```json
{
"total" : 1,
"activities": [
// activities list
]
}
```
## `GET /api/pleroma/admin/instances/:instance/statuses`
......@@ -300,7 +311,18 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
- *optional* `with_reblogs`: `true`/`false` – allows to see reblogs (default is false)
- Response:
- On failure: `Not found`
- On success: JSON array of instance's latest statuses
- On success: JSON, where:
- `total`: total count of the statuses for the instance
- `activities`: list of the statuses for the instance
```json
{
"total" : 1,
"activities": [
// activities list
]
}
```
## `GET /api/pleroma/admin/statuses`
......
......@@ -16,6 +16,12 @@ Adding the parameter `reply_visibility` to the public and home timelines queries
Adding the parameter `instance=lain.com` to the public timeline will show only statuses originating from `lain.com` (or any remote instance).
Home, public, hashtag & list timelines accept these parameters:
- `only_media`: show only statuses with media attached
- `local`: show only local statuses
- `remote`: show only remote statuses
## Statuses
- `visibility`: has additional possible values `list` and `local` (for local-only statuses)
......@@ -54,6 +60,23 @@ The `id` parameter can also be the `nickname` of the user. This only works in th
- `/api/v1/accounts/:id`
- `/api/v1/accounts/:id/statuses`
`/api/v1/accounts/:id/statuses` endpoint accepts these parameters:
- `pinned`: include only pinned statuses
- `tagged`: with tag
- `only_media`: include only statuses with media attached
- `with_muted`: include statuses/reactions from muted accounts
- `exclude_reblogs`: exclude reblogs
- `exclude_replies`: exclude replies
- `exclude_visibilities`: exclude visibilities
Endpoints which accept `with_relationships` parameter:
- `/api/v1/accounts/:id`
- `/api/v1/accounts/:id/followers`
- `/api/v1/accounts/:id/following`
- `/api/v1/mutes`
Has these additional fields under the `pleroma` object:
- `ap_id`: nullable URL string, ActivityPub id of the user
......
......@@ -242,6 +242,13 @@ def run(["gen" | rest]) do
rum_enabled: rum_enabled
)
config_dir = Path.dirname(config_path)
psql_dir = Path.dirname(psql_path)
[config_dir, psql_dir, static_dir, uploads_dir]
|> Enum.reject(&File.exists?/1)
|> Enum.map(&File.mkdir_p!/1)
shell_info("Writing config to #{config_path}.")
File.write(config_path, result_config)
......@@ -275,10 +282,6 @@ defp write_robots_txt(static_dir, indexable, template_dir) do
indexable: indexable
)
unless File.exists?(static_dir) do
File.mkdir_p!(static_dir)
end
robots_txt_path = Path.join(static_dir, "robots.txt")
if File.exists?(robots_txt_path) do
......
defmodule Mix.Tasks.Pleroma.OpenapiSpec do
def run([path]) do
spec = Pleroma.Web.ApiSpec.spec(server_specific: false) |> Jason.encode!()
File.write(path, spec)
end
end
......@@ -113,11 +113,15 @@ def create(title, %User{} = creator) do
end
end
def follow(%Pleroma.List{following: following} = list, %User{} = followed) do
def follow(%Pleroma.List{id: id}, %User{} = followed) do
list = Repo.get(Pleroma.List, id)
%{following: following} = list
update_follows(list, %{following: Enum.uniq([followed.follower_address | following])})
end
def unfollow(%Pleroma.List{following: following} = list, %User{} = unfollowed) do
def unfollow(%Pleroma.List{id: id}, %User{} = unfollowed) do
list = Repo.get(Pleroma.List, id)
%{following: following} = list
update_follows(list, %{following: List.delete(following, unfollowed.follower_address)})
end
......
......@@ -146,6 +146,7 @@ defmodule Pleroma.User do
field(:inbox, :string)
field(:shared_inbox, :string)
field(:accepts_chat_messages, :boolean, default: nil)
field(:last_active_at, :naive_datetime)
embeds_one(
:notification_settings,
......@@ -2030,6 +2031,15 @@ def local_nickname(nickname_or_mention) do
|> hd()
end
def full_nickname(%User{} = user) do
if String.contains?(user.nickname, "@") do
user.nickname
else
%{host: host} = URI.parse(user.ap_id)
user.nickname <> "@" <> host
end
end
def full_nickname(nickname_or_mention),
do: String.trim_leading(nickname_or_mention, "@")
......@@ -2444,4 +2454,19 @@ def sanitize_html(%User{} = user, filter) do
def get_host(%User{ap_id: ap_id} = _user) do
URI.parse(ap_id).host
end
def update_last_active_at(%__MODULE__{local: true} = user) do
user
|> cast(%{last_active_at: NaiveDateTime.utc_now()}, [:last_active_at])
|> update_and_set_cache()
end
def active_user_count(weeks \\ 4) do
active_after = Timex.shift(NaiveDateTime.utc_now(), weeks: -weeks)
__MODULE__
|> where([u], u.last_active_at >= ^active_after)
|> where([u], u.local == true)
|> Repo.aggregate(:count)
end
end
......@@ -591,7 +591,21 @@ def fetch_user_abstract_activities(user, reading_user, params \\ %{}) do
|> Enum.reverse()
end
def fetch_user_activities(user, reading_user, params \\ %{}) do
def fetch_user_activities(user, reading_user, params \\ %{})
def fetch_user_activities(user, reading_user, %{total: true} = params) do
result = fetch_activities_for_user(user, reading_user, params)
Keyword.put(result, :items, Enum.reverse(result[:items]))
end
def fetch_user_activities(user, reading_user, params) do
user
|> fetch_activities_for_user(reading_user, params)
|> Enum.reverse()
end
defp fetch_activities_for_user(user, reading_user, params) do
params =
params
|> Map.put(:type, ["Create", "Announce"])
......@@ -616,10 +630,20 @@ def fetch_user_activities(user, reading_user, params \\ %{}) do
}
|> user_activities_recipients()
|> fetch_activities(params, pagination_type)
|> Enum.reverse()
end
def fetch_statuses(reading_user, %{total: true} = params) do
result = fetch_activities_for_reading_user(reading_user, params)
Keyword.put(result, :items, Enum.reverse(result[:items]))
end
def fetch_statuses(reading_user, params) do
reading_user
|> fetch_activities_for_reading_user(params)
|> Enum.reverse()
end
defp fetch_activities_for_reading_user(reading_user, params) do
params = Map.put(params, :type, ["Create", "Announce"])
%{
......@@ -628,7 +652,6 @@ def fetch_statuses(reading_user, params) do
}
|> user_activities_recipients()
|> fetch_activities(params, :offset)
|> Enum.reverse()
end
defp user_activities_recipients(%{godmode: true}), do: []
......@@ -735,6 +758,12 @@ defp restrict_local(query, %{local_only: true}) do
defp restrict_local(query, _), do: query
defp restrict_remote(query, %{remote: true}) do
from(activity in query, where: activity.local == false)
end
defp restrict_remote(query, _), do: query
defp restrict_actor(query, %{actor_id: actor_id}) do
from(activity in query, where: activity.actor == ^actor_id)
end
......@@ -1111,6 +1140,7 @@ def fetch_activities_query(recipients, opts \\ %{}) do
|> restrict_tag_all(opts)
|> restrict_since(opts)
|> restrict_local(opts)
|> restrict_remote(opts)
|> restrict_actor(opts)
|> restrict_type(opts)
|> restrict_state(opts)
......
......@@ -85,17 +85,18 @@ def list_instance_statuses(conn, %{"instance" => instance} = params) do
with_reblogs = params["with_reblogs"] == "true" || params["with_reblogs"] == true
{page, page_size} = page_params(params)
activities =
result =
ActivityPub.fetch_statuses(nil, %{
instance: instance,
limit: page_size,
offset: (page - 1) * page_size,
exclude_reblogs: not with_reblogs
exclude_reblogs: not with_reblogs,
total: true
})
conn
|> put_view(AdminAPI.StatusView)
|> render("index.json", %{activities: activities, as: :activity})
|> render("index.json", %{total: result[:total], activities: result[:items], as: :activity})
end
def list_user_statuses(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname} = params) do
......@@ -105,18 +106,19 @@ def list_user_statuses(%{assigns: %{user: admin}} = conn, %{"nickname" => nickna
with %User{} = user <- User.get_cached_by_nickname_or_id(nickname, for: admin) do
{page, page_size} = page_params(params)
activities =
result =
ActivityPub.fetch_user_activities(user, nil, %{
limit: page_size,
offset: (page - 1) * page_size,
godmode: godmode,
exclude_reblogs: not with_reblogs,
pagination_type: :offset
pagination_type: :offset,
total: true
})
conn
|> put_view(AdminAPI.StatusView)
|> render("index.json", %{activities: activities, as: :activity})
|> render("index.json", %{total: result[:total], activities: result[:items], as: :activity})
else
_ -> {:error, :not_found}
end
......
......@@ -13,6 +13,10 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
defdelegate merge_account_views(user), to: AdminAPI.AccountView
def render("index.json", %{total: total} = opts) do
%{total: total, activities: safe_render_many(opts.activities, __MODULE__, "show.json", opts)}
end
def render("index.json", opts) do
safe_render_many(opts.activities, __MODULE__, "show.json", opts)
end
......
......@@ -11,10 +11,10 @@ defmodule Pleroma.Web.ApiSpec do
@behaviour OpenApi
@impl OpenApi
def spec do
def spec(opts \\ []) do
%OpenApi{
servers:
if Phoenix.Endpoint.server?(:pleroma, Endpoint) do
if opts[:server_specific] do
[
# Populate the Server info from a phoenix endpoint
OpenApiSpex.Server.from_endpoint(Endpoint)
......@@ -23,9 +23,25 @@ def spec do
[]
end,
info: %OpenApiSpex.Info{
title: "Pleroma",
description: Application.spec(:pleroma, :description) |> to_string(),
version: Application.spec(:pleroma, :vsn) |> to_string()
title: "Pleroma API",
description: """
This is documentation for client Pleroma API. Most of the endpoints and entities come
from Mastodon API and have custom extensions on top.
While this document aims to be a complete guide to the client API Pleroma exposes,
the details are still being worked out. Some endpoints may have incomplete or poorly worded documentation.
You might want to check the following resources if something is not clear:
- [Legacy Pleroma-specific endpoint documentation](https://docs-develop.pleroma.social/backend/development/API/pleroma_api/)
- [Mastodon API documentation](https://docs.joinmastodon.org/client/intro/)
- [Differences in Mastodon API responses from vanilla Mastodon](https://docs-develop.pleroma.social/backend/development/API/differences_in_mastoapi_responses/)
Please report such occurences on our [issue tracker](https://git.pleroma.social/pleroma/pleroma/-/issues). Feel free to submit API questions or proposals there too!
""",
version: Application.spec(:pleroma, :vsn) |> to_string(),
extensions: %{
# Logo path should be picked so that the path exists both on Pleroma instances and on api.pleroma.social
"x-logo": %{"url" => "/static/logo.svg", "altText" => "Pleroma logo"}
}
},
# populate the paths from a phoenix router
paths: OpenApiSpex.Paths.from_router(Router),
......@@ -45,15 +61,73 @@ def spec do
authorizationUrl: "/oauth/authorize",
tokenUrl: "/oauth/token",
scopes: %{
"read" => "read",
"write" => "write",
"follow" => "follow",
"push" => "push"
# TODO: Document granular scopes
"read" => "Read everything",
"write" => "Write everything",
"follow" => "Manage relationships",
"push" => "Web Push API subscriptions"
}
}
}
}
}
},
extensions: %{
# Redoc-specific extension, every time a new tag is added it should be reflected here,
# otherwise it won't be shown.
"x-tagGroups": [
%{
"name" => "Accounts",
"tags" => ["Account actions", "Retrieve account information", "Scrobbles"]
},
%{
"name" => "Administration",
"tags" => [
"Chat administration",
"Emoji packs",
"Frontend managment",
"Instance configuration",
"Instance documents",
"Invites",
"MediaProxy cache",
"OAuth application managment",
"Report managment",
"Relays",
"Status administration"
]
},
%{"name" => "Applications", "tags" => ["Applications", "Push subscriptions"]},
%{
"name" => "Current account",
"tags" => [
"Account credentials",
"Backups",
"Blocks and mutes",
"Data import",
"Domain blocks",
"Follow requests",
"Mascot",
"Markers",
"Notifications"
]
},
%{"name" => "Instance", "tags" => ["Custom emojis"]},
%{"name" => "Messaging", "tags" => ["Chats", "Conversations"]},
%{
"name" => "Statuses",
"tags" => [
"Emoji reactions",
"Lists",
"Polls",
"Timelines",
"Retrieve status information",
"Scheduled statuses",
"Search",
"Status actions"
]
},
%{"name" => "Miscellaneous", "tags" => ["Reports", "Suggestions"]}
]
}
}
# discover request/response schemas from path specs
......
......@@ -26,7 +26,7 @@ def open_api_operation(action) do
@spec create_operation() :: Operation.t()
def create_operation do
%Operation{