MRF rejection on fetched objects apparently not recorded, causing DDoS
Specifically, this can create an accidental DDoS when there is a very long thread with a lot of interactions that is rejected when other servers try to fetch it. If the attempt/rejection are not recorded, then the various servers will attempt to refetch every time they see a reference to the rejected object.
In this specific case, we have the lengthy hellthread, our hellthread policy delists at 32 and rejects at 256, somewhat more liberal than typical configurations. Some activities occurred that caused other servers to attempt to fetch objects in that thread, and this caused a flood of requests, with some servers attempting to re-fetch the same few objects several times per second over the course of about 30 minutes, in some cases attempting to fetch the same object more than a thousand times. For example, SPC tried to fetch /objects/5e64a211-12a3-4c70-aeac-96b8346e7e5f
1,242 times, rejected it (probably due to their hellthread policy having much lower limits), and then promptly discarded the object, forgot about it, and tried to fetch it again. Across other servers, there were 10,129 attempts to fetch that object; there were 120 objects that had at least 100 requests from any single server, Pleroma dutifully attempted to load the object from the database each time, and we fell over. Rate-limiting wasn't helpful, because the requests were coming from different servers and hitting different URLs. (Other servers that had people that posted in the thread might have seen similar, but I think the majority of the posts in the thread were local.)
I might be wrong about the cause being MRF rejections when fetching posts, but I can't account for it otherwise; the servers are running different versions of Pleroma, they're not run by the same people, etc.
(There is a workaround, but it's somewhat hacky: it was to grep the logs for any path under /objects/ that a remote server had tried to fetch more than 100 times in an hour, then generate a long list of of lines like location /objects/fb8c859a-f193-49cd-ae8d-afe8967163e4 {return 404;}
and include the result in the nginx configuration, and temporarily, we're somewhat aggressively rate-limiting /objects/
.)