robots_txt.md 652 Bytes
Newer Older
feld's avatar
feld committed
1
# Managing robots.txt
Ilja's avatar
Ilja committed
2
3
4

{! backend/administration/CLI_tasks/general_cli_task_info.include !}

feld's avatar
feld committed
5
## Generate a new robots.txt file and add it to the static directory
Ilja's avatar
Ilja committed
6
7
8
9
10

The `robots.txt` that ships by default is permissive. It allows well-behaved search engines to index all of your instance's URIs.

If you want to generate a restrictive `robots.txt`, you can run the following mix task. The generated `robots.txt` will be written in your instance [static directory](../../../configuration/static_dir/).

rinpatch's avatar
rinpatch committed
11
=== "OTP"
Ilja's avatar
Ilja committed
12

rinpatch's avatar
rinpatch committed
13
14
15
16
17
18
19
20
21
    ```sh
    ./bin/pleroma_ctl robots_txt disallow_all
    ```

=== "From Source"

    ```sh
    mix pleroma.robots_txt disallow_all
    ```