Skip to content
Snippets Groups Projects
Verified Commit 97dc6336 authored by Alexander Strizhakov's avatar Alexander Strizhakov
Browse files

dump only settings needed for reboot on database config update

parent dddebee0
Branches fix/remote-follow-errormessage
No related tags found
No related merge requests found
......@@ -60,5 +60,8 @@
)
end
if File.exists?("./config/dev.for_reboot.exs"),
do: import_config("dev.for_reboot.exs")
if File.exists?("./config/dev.exported_from_db.secret.exs"),
do: import_config("dev.exported_from_db.secret.exs")
......@@ -65,5 +65,8 @@
# which should be versioned separately.
import_config "prod.secret.exs"
if File.exists?("./config/prod.for_reboot.exs"),
do: import_config("prod.for_reboot.exs")
if File.exists?("./config/prod.exported_from_db.secret.exs"),
do: import_config("prod.exported_from_db.secret.exs")
......@@ -21,10 +21,15 @@
IO.puts(warning)
end
exported_config =
config_path
|> Path.dirname()
|> Path.join("prod.exported_from_db.secret.exs")
config_path_dir = Path.dirname(config_path)
for_reboot_config = Path.join(config_path_dir, "prod.for_reboot.exs")
if File.exists?(for_reboot_config) do
import_config for_reboot_config
end
exported_config = Path.join(config_path_dir, "prod.exported_from_db.secret.exs")
if File.exists?(exported_config) do
import_config exported_config
......
......@@ -6,15 +6,15 @@ config :pleroma, configurable_from_database: true
```
## How it works
Settings are stored in database and are applied in `runtime` after each change. Most of the settings take effect immediately, except some, which need instance reboot. These settings are needed in `compile time`, that's why settings are duplicated to the file.
Settings are stored in database and are applied in `runtime` after each change. Most of the settings take effect immediately, except some, which need instance reboot. These settings are needed in `compile time`, that's why these settings are duplicated to the file.
File with duplicated settings is located in `config/{env}.exported_from_db.exs` if pleroma is runned from source. For prod env it will be `config/prod.exported_from_db.exs`.
File with duplicated settings is located in `config/{env}.for_reboot.exs` if pleroma is runned from source. For prod env it will be `config/prod.for_reboot.exs`.
For releases: `/etc/pleroma/prod.exported_from_db.secret.exs` or `PLEROMA_CONFIG_PATH/prod.exported_from_db.exs`.
For releases: `/etc/pleroma/prod.for_reboot.exs` or `PLEROMA_CONFIG_PATH/prod.for_reboot.exs`.
## How to set it up
You need to migrate your existing settings to the database. This task will migrate only added by user settings.
For example you add settings to `prod.secret.exs` file, only these settings will be migrated to database. For release it will be `/etc/pleroma/config.exs` or `PLEROMA_CONFIG_PATH`.
For example you add settings to `prod.secret.exs` file, only these settings will be migrated to the database. For releases path will be `/etc/pleroma/config.exs` or `PLEROMA_CONFIG_PATH`.
You can do this with mix task (all config files will remain untouched):
```sh tab="OTP"
......@@ -25,11 +25,11 @@ You can do this with mix task (all config files will remain untouched):
mix pleroma.config migrate_to_db
```
Now you can change settings in admin interface. After each save, settings from database are duplicated to the `config/{env}.exported_from_db.exs` file.
Now you can change settings in admin interface. After each save, `compile time` settings from database are duplicated to the `config/{env}.for_reboot.exs` file.
<span style="color:red">**ATTENTION**</span>
**<span style="color:red">Be careful while changing the settings. Every inaccurate configuration change can break the federation or the instance load.</span>**
**<span style="color:red">Be careful while changing the settings. Every inaccurate configuration change can break federation or the instance load.</span>**
*Compile time settings, which require instance reboot and can break instance loading:*
- all settings inside these keys:
......@@ -40,9 +40,11 @@ Now you can change settings in admin interface. After each save, settings from d
- `:proxy_remote` in `Pleroma.Upload`
- `:upload_limit` in `:instance`
## How to dump settings from database to file
## How to dump all settings from database to file
All settings will be dumped to `config/{env}.expoerted_from_db.secret.exs` if pleroma is runned from source.
For releases: `/etc/pleroma/prod.exported_from_db.secret.exs` or `PLEROMA_CONFIG_PATH/prod.exported_from_db.secret.exs`.
*Adding `-d` flag will delete migrated settings from database table.*
*Adding `-d` flag will delete migrated settings from database `config` table.*
```sh tab="OTP"
./bin/pleroma_ctl config migrate_from_db [-d]
......@@ -52,20 +54,20 @@ Now you can change settings in admin interface. After each save, settings from d
mix pleroma.config migrate_from_db [-d]
```
## How to completely remove it
1. Truncate or delete all values from `config` table
```sql
TRUNCATE TABLE config;
```
2. Delete `config/{env}.exported_from_db.exs`.
2. Delete `config/{env}.for_reboot.exs` and `config/{env}.exported_from_db.secret.exs`.
For `prod` env:
```bash
cd /opt/pleroma
cp config/prod.exported_from_db.exs config/exported_from_db.back
rm -rf config/prod.exported_from_db.exs
cp config/prod.exported_from_db.secret.exs config/exported_from_db.back
rm -rf config/prod.for_reboot.exs
rm -rf config/prod.exported_from_db.secret.exs
```
*If you don't want to backup settings, you can skip step with `cp` command.*
......
......@@ -18,7 +18,7 @@ mix pleroma.config migrate_to_db
## Transfer config from DB to `config/env.exported_from_db.secret.exs`
To delete transfered settings from database optional flag `-d` can be used. <env> is `prod` by default.
To delete transfered settings from database optional flag `-d` can be used. `<env>` is `prod` by default.
```sh tab="OTP"
./bin/pleroma_ctl config migrate_from_db [--env=<env>] [-d]
```
......
......@@ -13,6 +13,17 @@ defmodule Mix.Tasks.Pleroma.Config do
@shortdoc "Manages the location of the config"
@moduledoc File.read!("docs/administration/CLI_tasks/config.md")
@reboot_time_keys [
{:pleroma, :hackney_pools},
{:pleroma, :chat}
]
@reboot_time_subkeys [
{:pleroma, Pleroma.Captcha, [:seconds_valid]},
{:pleroma, Pleroma.Upload, [:proxy_remote]},
{:pleroma, :instance, [:upload_limit]}
]
def run(["migrate_to_db"]) do
start_pleroma()
migrate_to_db()
......@@ -50,6 +61,20 @@ def migrate_to_db(file_path \\ nil) do
end
end
@spec dump_reboot_settings() :: :ok
def dump_reboot_settings do
with config_path <- config_path(),
file <- open_file_with_header(config_path) do
ConfigDB
|> Repo.all()
|> Enum.each(&filter_and_write(&1, file))
close_and_format_file(file, config_path)
:ok
end
end
defp do_migrate_to_db(config_file) do
if File.exists?(config_file) do
custom_config =
......@@ -78,30 +103,50 @@ defp create(group, settings) do
shell_info("Settings for group :#{group} migrated.")
end
defp migrate_from_db(opts) do
if Pleroma.Config.get([:configurable_from_database]) do
env = opts[:env] || "prod"
defp config_path do
:env
|> Pleroma.Config.get()
|> to_string()
|> config_path("for_reboot")
end
config_path =
if Pleroma.Config.get(:release) do
:config_path
|> Pleroma.Config.get()
|> Path.dirname()
else
"config"
end
|> Path.join("#{env}.exported_from_db.secret.exs")
defp config_path(opts) when is_list(opts) do
(opts[:env] || "prod")
|> config_path()
end
file = File.open!(config_path, [:write, :utf8])
defp config_path(env, tail \\ "exported_from_db.secret") when is_binary(env) do
if Pleroma.Config.get(:release) do
:config_path
|> Pleroma.Config.get()
|> Path.dirname()
else
"config"
end
|> Path.join("#{env}.#{tail}.exs")
end
IO.write(file, config_header())
defp open_file_with_header(config_path) do
file = File.open!(config_path, [:write, :utf8])
IO.write(file, config_header())
file
end
ConfigDB
|> Repo.all()
|> Enum.each(&write_and_delete(&1, file, opts[:delete]))
defp close_and_format_file(file, config_path) do
:ok = File.close(file)
System.cmd("mix", ["format", config_path])
end
:ok = File.close(file)
System.cmd("mix", ["format", config_path])
defp migrate_from_db(opts) do
if Pleroma.Config.get([:configurable_from_database]) do
with config_path <- config_path(opts),
file <- open_file_with_header(config_path) do
ConfigDB
|> Repo.all()
|> Enum.each(&write_and_delete(&1, file, opts[:delete]))
close_and_format_file(file, config_path)
end
else
migration_error()
end
......@@ -121,6 +166,27 @@ defp config_header, do: "use Mix.Config\r\n\r\n"
defp read_file(config_file), do: Mix.Config.eval!(config_file)
end
defp filter_and_write(config, file) do
group = ConfigDB.from_string(config.group)
key = ConfigDB.from_string(config.key)
value = ConfigDB.from_binary(config.value)
if Enum.any?(@reboot_time_keys, fn {g, k} -> g == group and k == key end) do
write(config, file)
else
with {_g, _k, subkeys} <-
Enum.find(@reboot_time_subkeys, fn {g, k, _subkeys} ->
g == group and k == key and Keyword.keyword?(value)
end),
for_dump when for_dump != [] <- Keyword.take(value, subkeys) do
IO.write(
file,
"config #{config.group}, #{config.key}, #{inspect(for_dump, limit: :infinity)}\r\n\r\n"
)
end
end
end
defp write_and_delete(config, file, delete?) do
config
|> write(file)
......
......@@ -3,8 +3,6 @@
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Config.Loader do
@paths ["config/config.exs", "config/#{Mix.env()}.exs"]
@reject_keys [
Pleroma.Repo,
Pleroma.Web.Endpoint,
......@@ -35,8 +33,8 @@ defp do_merge(conf1, conf2), do: Mix.Config.merge(conf1, conf2)
def load_and_merge do
all_paths =
if Pleroma.Config.get(:release),
do: @paths ++ ["config/releases.exs"],
else: @paths
do: ["config/config.exs", "config/releases.exs"],
else: ["config/config.exs"]
all_paths
|> Enum.map(&load(&1))
......
......@@ -892,11 +892,7 @@ def config_update(conn, %{"configs" => configs}) do
Pleroma.Config.TransferTask.load_and_update_env(deleted)
Mix.Tasks.Pleroma.Config.run([
"migrate_from_db",
"--env",
to_string(Pleroma.Config.get(:env))
])
Mix.Tasks.Pleroma.Config.dump_reboot_settings()
conn
|> put_view(ConfigView)
......
......@@ -51,7 +51,7 @@ test "settings are migrated to db" do
assert ConfigDB.from_binary(config3.value) == :info
end
describe "with deletion temp file" do
describe "migrate_from_db" do
setup do
temp_file = "config/temp.exported_from_db.secret.exs"
......@@ -87,7 +87,27 @@ test "settings are migrated to file and deleted from db", %{temp_file: temp_file
assert file =~ "config :quack, :level, :info"
end
test "load a settings with large values and pass to file", %{temp_file: temp_file} do
test "without deletion from db" do
ConfigDB.create(%{
group: ":pleroma",
key: ":setting_first",
value: [key: "value", key2: ["Activity"]]
})
ConfigDB.create(%{
group: ":pleroma",
key: ":setting_second",
value: [key: "value2", key2: [Repo]]
})
ConfigDB.create(%{group: ":quack", key: ":level", value: :info})
Mix.Tasks.Pleroma.Config.run(["migrate_from_db", "--env", "temp"])
assert length(Repo.all(ConfigDB)) == 3
end
test "loads settings with large values and pass them to file", %{temp_file: temp_file} do
ConfigDB.create(%{
group: ":pleroma",
key: ":instance",
......@@ -173,4 +193,76 @@ test "load a settings with large values and pass to file", %{temp_file: temp_fil
"#{header}\n\nconfig :pleroma, :instance,\n name: \"Pleroma\",\n email: \"example@example.com\",\n notify_email: \"noreply@example.com\",\n description: \"A Pleroma instance, an alternative fediverse server\",\n limit: 5000,\n chat_limit: 5000,\n remote_limit: 100_000,\n upload_limit: 16_000_000,\n avatar_upload_limit: 2_000_000,\n background_upload_limit: 4_000_000,\n banner_upload_limit: 4_000_000,\n poll_limits: %{\n max_expiration: 31_536_000,\n max_option_chars: 200,\n max_options: 20,\n min_expiration: 0\n },\n registrations_open: true,\n federating: true,\n federation_incoming_replies_max_depth: 100,\n federation_reachability_timeout_days: 7,\n federation_publisher_modules: [Pleroma.Web.ActivityPub.Publisher],\n allow_relay: true,\n rewrite_policy: Pleroma.Web.ActivityPub.MRF.NoOpPolicy,\n public: true,\n quarantined_instances: [],\n managed_config: true,\n static_dir: \"instance/static/\",\n allowed_post_formats: [\"text/plain\", \"text/html\", \"text/markdown\", \"text/bbcode\"],\n mrf_transparency: true,\n mrf_transparency_exclusions: [],\n autofollowed_nicknames: [],\n max_pinned_statuses: 1,\n no_attachment_links: true,\n welcome_user_nickname: nil,\n welcome_message: nil,\n max_report_comment_size: 1000,\n safe_dm_mentions: false,\n healthcheck: false,\n remote_post_retention_days: 90,\n skip_thread_containment: true,\n limit_to_local_content: :unauthenticated,\n user_bio_length: 5000,\n user_name_length: 100,\n max_account_fields: 10,\n max_remote_account_fields: 20,\n account_field_name_length: 512,\n account_field_value_length: 2048,\n external_user_synchronization: true,\n extended_nickname_format: true,\n multi_factor_authentication: [\n totp: [digits: 6, period: 30],\n backup_codes: [number: 2, length: 6]\n ]\n"
end
end
describe "dump_reboot_settings/0" do
setup do
temp_file = "config/test.for_reboot.exs"
on_exit(fn ->
:ok = File.rm(temp_file)
end)
{:ok, temp_file: temp_file}
end
test "dumps only reboot time keys", %{temp_file: temp_file} do
ConfigDB.create(%{
group: ":pleroma",
key: ":instance",
value: [key: "value", key2: ["Activity"]]
})
ConfigDB.create(%{
group: ":pleroma",
key: ":hackney_pools",
value: "some_value"
})
ConfigDB.create(%{
group: ":pleroma",
key: ":chat",
value: [key: "value2"]
})
Mix.Tasks.Pleroma.Config.dump_reboot_settings()
{:ok, file} = File.read(temp_file)
assert file =~ "config :pleroma, :hackney_pools"
assert file =~ "config :pleroma, :chat"
refute file =~ "config :pleroma, :instance"
end
test "dumps only reboot time subkeys", %{temp_file: temp_file} do
ConfigDB.create(%{
group: ":pleroma",
key: ":instance",
value: [upload_limit: :limit, another_key: :value]
})
ConfigDB.create(%{
group: ":pleroma",
key: "Pleroma.Captcha",
value: [seconds_valid: 60, another_key: :value]
})
ConfigDB.create(%{
group: ":pleroma",
key: "Pleroma.Upload",
value: [proxy_remote: :remote, another_key: :value]
})
Mix.Tasks.Pleroma.Config.dump_reboot_settings()
{:ok, file} = File.read(temp_file)
assert file =~ "config :pleroma, :instance, upload_limit: :limit"
assert file =~ "config :pleroma, Pleroma.Captcha, seconds_valid: 60"
assert file =~ "config :pleroma, Pleroma.Upload, proxy_remote: :remote"
refute file =~ "another_key"
refute file =~ ":value"
end
end
end
......@@ -2043,7 +2043,7 @@ test "POST /api/pleroma/admin/config error", %{conn: conn} do
Application.delete_env(:pleroma, Pleroma.Captcha.NotReal)
Application.put_env(:pleroma, :http, http)
Application.put_env(:tesla, :adapter, Tesla.Mock)
:ok = File.rm("config/test.exported_from_db.secret.exs")
:ok = File.rm("config/test.for_reboot.exs")
end)
end
......@@ -2977,6 +2977,7 @@ test "proxy tuple ip", %{conn: conn} do
end
test "transfer settings to DB and to file", %{conn: conn} do
on_exit(fn -> :ok = File.rm("config/test.exported_from_db.secret.exs") end)
assert Repo.all(Pleroma.ConfigDB) == []
Mix.Tasks.Pleroma.Config.migrate_to_db("test/fixtures/config/temp.secret.exs")
assert Repo.aggregate(Pleroma.ConfigDB, :count, :id) > 0
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment